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ABSTRACT 
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was introduced. Evaluations were based on the benefit of 
information transferred to end users. The feasibility of this 
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The concept of using subjective evaluations to measure 
effectiveness in a more timely manner was also introduced and 
demonstrated to be a useful approach to estimate ROI with rea- 
sonable accuracy. 

An automated information system using the Statistical 
Package for the Social Sciences was also expanded in an opera- 
tional environment to accommodate the data processing require- 
ments of an effectiveness measuring system for the Facilities 
Engineering Support Office, a service organization of the 
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iw INTRODUCTION 


A. PERFORMANCE MEASURES 
The need to measure the performance of programmed efforts 

has always been of -concern to the manager, whether he is lo- 
cated at the operational or headquarters level. In these 
days of limited resources, the need has become more critical. 
Two common measures of performance are efficiency and effec- 
tiveness. Drucker [Ref. l]. in his discussion of the differ- 
ence between efficiency and effectiveness, stated: 

"Effectiveness is the function of success--efficiency is 

a minimum condition for survival after success has been 

achieved. Efficiency is concerned with doing things 

right. Effectiveness is doing the right things. 

Fr EN CNEY CONncerns itself with the input of effort into 

all areas of activity. Effectiveness, however, Starts 

out with the realization that in business, as in any other 

social organism, lO or l5 percent of the phenomena--—such 

as products, orders, customers, markets, or people-- 

produce 80 to 90 percent of the results. The other 85 

to 90 percent of the phenomena, no matter how efficiently 

taken care of, produce nothing but costs (which are always 

proportionate to transactions, that is busy-ness)." 

This study concentrates on efforts to measure effective- 

ness. However, the effectiveness of performance in certain 
areas of operations has essentially eluded evaluation efforts. 


This is generally true in the area of research and 


development [Ref. 2]. 


B. MEASURING EFFECTIVENESS 
In order to evaluate the effectiveness of research efforts, 
the manager must first decide what is going to be evaluated. 


In the case of a particular program, the purpose of that 
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program must be clearly stated to allow a definition of what 
is valuable. The manager must then either estimate the 
potential value or benefit of future events, assuming that 
those events will in fact take place, or wait until the 
results of the research are known. At some later date, if 
the results are committed to practice, he will assess the 
benefit that has been realized. The former approach is 
frought with subjectivity. The later approach entails an 
unacceptable waiting period, as well as a significant degree 
of subjectivity. Neither approach is desirable. 

Coupled with the inherent problems of subjectivity and 
timeliness, is the question of which evaluation techniques 
to use to measure effectiveness. In the area of research 
and development, they do not exist, for the most part, 
although progress has been made in certain limited areas. 
Finally, the measuring system must not require an inordin- 
ately large level of effort compared to the effort being 
evaluated. A small organization or a small program should, 
therefore, only allocate a limited amount of resources to 
measuring effectiveness. Managers faced with these obstacles 
and constraints are often inclined to abandon hope of quanti- 
tatively measuring program effectiveness. 

An alternative to abandoning hope, is to consider 
adapting those techniques for evaluating benefit which are 
workable, and through the use of a sampling approach, obtain 
sufficient data to estimate total benefit at a level of 


accuracy commensurate with the resources available for 


ri 





measuring effectiveness. The success of such an effort 
offers the manager a capability that does not currently 


exist, at an affordable price. 


C. PURPOSE OF THE STUDY 

The objective of this study is to develop an effective- 
ness measuring system based on sampling procedures and to 
demonstrate that it is a feasible approach to meeting manage- 
ment requirements for information on the effectiveness of a 
small service program. Such a system will provide a viable 
alternative in those situations where the lack of sufficient 
resources, or the relative magnitude of the program effort, 
precludes the accomplishment of full-scale evaluation efforts. 
Specifically, a previously developed evaluation technique 
will be optimized and applied to a selected number of indivi- 
dual service requests to provide the basis for estimating 
the annual return on investment (ROI) for the whole service 
program. It will be demonstrated that the data resulting 
from the evaluation of selected individual asssistance 
requests, when processed by a computer-aided management 
information system, provides a reasonably accurate and econ- 
omically affordable basis for estimating the annual ROI of 
resources expended by the Navy's Civil Engineering Laboratory 
in executing the Naval Facilities Engineering Command's 


Assistance Program. 
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II. BACKGROUND 


A. NAVAL FACILITIES ENGINEERING COMMAND 

The Naval Facilities Engineering Command (NAVFAC) provides 
engineering, material and equipment support to the Chief of 
Naval Operations, the Operating Forces of the Navy, the 
Marine Corps, components of the Naval Material Command, and 
other offices [Ref. 3]. NAVFAC's Research Program is directed 
primarily towards items of new or improved materials, equip- 
ment or engineering techniques which will significantly im- 
prove solutions to specific engineering problems pertaining 
to the technical planning, design, construction, operation 
and maintenance of the shore activities and fixed surface 
and subsurface structures of the Navy [Ref. 3]. 

NAVFAC's Research Program is specifically administered 
by the Assistant Commander for Research and Development 
(Code 03). NAVFAC Code 03 therefore has the responsibility 
to ensure that the output of Research and Development (R&D) 
is transferred to shore activities of the Navy and to ensure 
that maximum effectiveness is achieved from R&D investments. 

NAVFAC's link to the shore activities is primarily 
Garona 5 Engineering Field Divisions (EFDs), its Public 
Works Centers (PWCs), its construction program administered 
by Officers in Charge of Construction (OICCs) and Resident 
Officers in Charge of Construction (ROICCs), and Public 
Works Departments of individual shore activities. Figure l 


shows these organizational and technical relationships. 


J. 
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A major portion of NAVFAC's research program is assigned 
to and executed by the Navy's Civil Engineering Laboratory 


(CEDI). 


B. CIVIL ENGINEERING LABORATORY 

The Civil Engineering Laboratory (CEL) is under the 
administrative control of the Naval Construction Battalion 
Center, Port Hueneme, California, and is the principal RDT&E 
Center for shore and fixed surface and subsurface ocean 
eS ana Or the Navy and Marine Corps Construction 
ար ०. १). The staff of CEL consists of approximately 
300 personnel, ३375 of which are professional engineers and 
scientists, representing a large variety of disciplines. 

Funding for CEL's FY-76 RDT&E program amounted to 33 
million dollars with approximately 70 percent of the program 
supported by Navy RDT&E funds. The program represented 
Bpproximately 250 individual efforts, called work units. One 
of these work units was RDT&E Assistance to Naval Shore 


Activities. 


C. ASSISTANCE PROGRAM 

The work unit RDT&E Assistance to Naval Shore Activities, 
referred to as the Assistance Program in the remainder of 
this report, was initiated at CEL in 32963 using NAVFAC 
Exploratory Development (6.2 category RDT&E) funds. The pur- 
pose of the Assistance Program is to provide quick-response 
service, upon request and at no cost to the requestor, dir- 


ectly to users located at NAVFAC field activities and Public 
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Works Departments of shore activities throughout the Naval 
Shore Establishment. These efforts normally take the form 

of short-term investigations, analyses and tests to determine 
the relative value and suitability of new materials, equipment, 
processes, and construction and maintenance procedures [Ref. 
5]. The key words in this statement are "to provide services 
upon request." Thus the Assistance Program is driven by the 
expressed needs of its users. 

In the process of assisting shore activities, CEL 
researchers provide field personnel with an early awareness 
of current R&D efforts and preliminary and final results that 
relate to their areas of expressed interest. This results in 
narrowing the gap between research and use by putting the 
results of research to work at the earliest practicable time 
to solve current problems being experienced by field personnel 
involved in facilities engineering matters. 

During FY-76, a review of Navy RDT&E programs led to a 
decision by the Director of Defense Research and Engineering 
that the use of Navy Exploratory Development funds was not 
appropriate for the provision of assistance to Naval shore 
activities. Accordingly, an alternative funding source had 
to be developed if the Assistance Program was to be continued 
a ter byY— 76. 

After reviewing the situation, NAVFAC management decided 
that there was sufficient benefit in providing assistance to 
Naval shore activities and that these efforts should be con- 


tinued using mission management funds. Although performance 
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measures applied to CEL assistance efforts related to activity 
and growth in usage, rather than effectiveness, this informa- 
tion was used as input to this NAVFAC decision. These per- 


formance measures are discussed in some detail in Appendix A. 


DIAS TETAS T ENGINEERING SUPPORT OFFICE 

The Facilities Engineering Support Office (FESO) was 
established by CEL and funded by NAVFAC to perform the func- 
tion of coordinating services and communications related to 
providing assistance to shore activities. This was a reali- 
zation of management desire to focus attention on this aspect 
of CEL's efforts and to ensure thet timely and effective 
laboratory support was provided in response to field needs. 
The function was placed in CEL's organization as a staff com- 
ponent of the Technical Director. This location reflects 
management interest and dedication to this service and allows 
for easy and direct access to CEL's primary resource--its 
people. The basic organization of CEL is shown in Figure 2. 
The primary FESO functions related to the Assistance Program 
Sresoutlıneg. in Figure 3. 

NAVFAC's decision to continue the Assistance Program 
utilizing mission management funds was based on performance 
measures of program activity mentioned earlier and other sub- 
jective judgements of the value of the program. Accordingly, 
the need to quantify the effectiveness of the Assistance Pro- 
gram became a matter of increased management interest and 


importance. Developing a system to measure the effectiveness 
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The functions of the Facilities Engineering Support 
Office (FESO) related to providing RDT&E Assistance to Naval 
Shore Activities are: 


l. Ensure that timely engineering support services are 
provided to NAVFAC and Public Works field activities 
throughout the Shore Establishment on facility engineering 
matters. 


2. Serve as a point of contact at CEL for liaison with 
field activities on facility engineering RDT&E matters. 


3. Search out, develop, and maintain a knowledge of the 
assistance needs of these field activities and provide 
them with a knowledge of CEL's capabilities and interest 
in meeting their needs. 


4. Act as a project coordinator on assistance projects 
for field activities to ensure timely and effective labor- 
atory responsiveness to these needs. 


5. Act as a focal point for correspondence relating to 
assistance to field activities, including promotion of 
direct contact between field personnel and individual mem- 
bers Of CEL’S technical staff. 


6. Effect utilization of currently available research 
results as a means of meeting the needs of field activities. 


7. Formulate plans for RDT&E support of field units by 
CEL where a solution is not currently available, including 
fostering the initiating of RDT&E task proposals and ob- 
taining field activity sponsorship of such RDT&E proposals. 


8. Study the spectra of assistance needs of the field 
activities, synthesize these needs and their solutions, 
and disseminate findings to these field activities [Ref. 4]. 


Figure 3. FESO Functions Related to The Assistance Program 
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of assistance efforts in terms of benefits to users would 
satisfy management requirements, and also provide valuable 


feedback information to guide overall FESO efforts. 
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HUT SCOEE OE THE WORK 


A. ASSUMPTIONS 

The basic assumption of this study is that a pressing 
need exists to obtain quantified information on the effective- 
ness of the Assistance Program on an on-going basis. 

A parallel assumption is that the expense of evaluating 
all assistance requests, on an on-going basis, is not war- 
ranted by the level of effort expended in providing assistance 
services. Even if it were justified, the effort required for 
such an evaluation would exceed the resources available to 
tne FESO. 

The prevailing rationale is that the primary purpose of 
the Assistance Program is to provide service to field users. 
Secondarily, the Assistance Program should be capable of pro- 
ducing an acceptable level of benefit for those users, which 
at a minimum would require that it pay for itself, or be self- 
Supporting. Of course, this must be taken figuratively, 

Since there is no exchange of funds between shore activities 
and CEL with regard to the Assistance Program. In the case 
of commercial businesses, the final criteria for benefit, 
Stated in terms of profit and loss, is return on investment. 
This criteria could be approximated for the Assistance Pro- 
gram by a measure of discounted present value of estimates 

of benefit provided by the users of the service. Accordingly, 


the Assistance Program should produce a minimum ROI of ]l.0. 
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The need to generate and process evaluation data with a 
relatively modest effort requires adoption of a computer- 
based effectiveness measuring system. However, the system 
must be operable with the data processing capability that is 
currently available to CEL. The system should be based on 
evaluating samples of the total requests received. The sys- 
tem should be flexible to allow evaluation of a variable 
number of assistance requests, or all requests for that 
matter. 

There are certain risks associated with the gathering of 
information related to the value of a particular effort. In 
the case of the Assistance Program these risks could material- 
ize in the form of negative attitudes toward CEL, toward the 
Assistance Program, or toward participating individuals. The 
creation of such negative attitudes would have a debilitating 
effect on the Assistance Program resulting in the decreased 
use Of CEL as a resource by potential users, and decreased 
interest in participating, or in recording participation, by 
CEL personnel. Therefore, only those assistance efforts 
which are judged to provide measurable benefit should be in- 


cluded in field evaluations of benefit. 


BE ZEIMITATIONS 

The measuring system estimates the effectiveness of the 
Assistance Program during a particular period of time. It 
does not measure actual benefit. This would require consider- 
able effort over extended periods of time. It does measure 


a collective expression of perceived benefit that users are 
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willing to make, stated in terms of estimated return on 
investment. 

Evaluating the benefits of a sample of the total requests 
received during a given period of time will not provide the 
basis for an exact value for the estimated total benefit 
during that period. The benefits of the individual assist- 
ance requests are normally distributed continuous variables. 
At best, these benefit values can be used to develop a reli- 
able estimator of the estimated total benefits that would 
accumulate during that period, with a known level of accuracy. 
For example, a significantly large benefit, compared to other 
benefits, might not be included in the sample and, therefore, 
would not influence the calculation of ROI. This would result 
in understating the range of estimated ROI. 

There is a risk in connection with the possible misuse 
of ROI data. It should be clearly understood that the esti- 
mated ROI for one period of time cannot be used to answer 
questions about the level of benefit achieved by the Assist- 
ance Program during other periods of time. 

There is a greater risk that ROI information developed to 
measure the effectiveness of the Assistance Program during 
various periods of time might be used in such away as to be 
inconsistent with the basic purpose of the program. Requests 
are received in random fashion, at the discretion of request- 
ing personnel at shore activities. Therefore, there is no 
logical basis for assuming that there should be a significant 


correlation between ROI estimated for different periods of 
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time, Or more importantly, that the ROI should continue to 
increase from year to year. The later assumption amounts to 
establishing a goal "to make an increasing larger profit, 
while providing a free service." If the primary purpose of 
the Assistance Program is to provide a service, at no cost 

to the user, serious consideration should be given to being 
content to see the value of ROI remain above 2.0, at which 
point the cost of providing the service is offset by benefits 
to its users. 

The benefits of the Assistance Program are many and varied. 
People in the field, at shore activities around the world 
receive the primary benefits. However, engineers and scien- 
tists at CEL share in the benefits of the Assistance Program 
too. 

Although the effectiveness of the program will be stated 
in terms of estimated ROI, it should be noted that these data 
are based only on that part of the total benefit that can be 
quantified with reasonable accuracy and credibility. Since 
a variety of additional benefits result from assistance 
efforts, it is worthwhile to list some of them, so they will 
not be overlooked until such time that techniques become 
available to quantify them. Some of these "additional bene- 


fits" are listed in Appendix B. 
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IV. METHODOLOGY 

A. SELECTION OF AN OPERATIONAL SYSTEM TO MEASURE 

EFFECTIVENESS 

Since its inception, the FESO has been interested in the 
evaluation of benefits as a measure of effectiveness, for use 
by CEL and NAVFAC. Efforts were undertaken at the Naval Post- 
graduate School to measure the effectiveness of the FY-72 and 
FY-73 assistance efforts [Refs. 6 and 7]. These efforts pro- 
duced a basic foundation for later efforts, although the 
results were not definitive in that they were based on less 
than adequate data gathering instruments, or questionnaires 
for gathering evaluation data, and insufficient coverage of 
the shore activities served. Later efforts by Hendrickson 
and Fisher [Ref. 8], to measure the effectiveness of the FY- 
74 Assistance Program, produced a more comprehensive question- 
naire and increased the evaluation efforts to include a 
larger postion of the activities served. A breakthrough also 
resulted in that the techniques for evaluating the worth of 
tangible and intangible benefits were advanced considerably. 
Although the study produced a workable system, the cost to 
measure program effectiveness on an on-going basis, using 
those techniques to evaluate each assistance request received 
by CEL, was considered prohibitively expensive in terms of 
available resources and in relation to the total cost of pro- 
viding assistance. Study results, which indicated a mean 
benefit of $2.72 for each dollar expended to provide assist- 


ance during FY-74, coupled with the potential cost of similar 
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measurement efforts, did relieve the immediate pressure for 
continuing efforts to measure return on investment for the 
program. This led to a decision in FY-75 to defer further 
effectiveness measuring for several years. The shift in fund- 
ing source to NAVFAC mission management funds created need 

for an operable system. It was believed that an operational 
system incorporating those techniques could be developed 
based on a modification of the Hendrickson and Fisher model. 
The cost barrier associated with evaluating all assistance 
requests received during a given period could be overcome by 
evaluating the benefit of a selected number of the requests 
and using these data as a basis for estimating the total bene- 


fit of all requests. 


B ELECTION OF A PERIOD TO STUDY 
During FY-76, CEL responded to 622 individual assistance 

requests from Naval shore activities. Although some thought 
was Originally given to including all those requests in the 
study, a review of the assistance activity during that year 
revealed several reasons why the study should focus on the 
last quarter of FY-76. The main reasons for selecting that 
period for study purposes were: 

l. The total of ]l45 assistance requests during the fourth- 


quarter was approximately one-fourth of the total number 
of requests received during the entire year. 


2. The total of ]l45 requests was approximately one-half of 
the number of requests treated in the Hendrickson and Fisher 
study of FY-74 assistance efforts. 


3. The 345 requests should represent a large enough samp- 
ling population for study purposes. 
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The system model is based on the evaluation of individual 
benefits to users. Although benefit is considered on a 
request-by-request basis, in the final analysis the model 
treats the entire Assistance Program as if it were a single 
CEL product resulting from a single investment, rather than 
a number of individual requests for assistance with their 
attendant costs and benefits. This follows the character of 
the R&D process, in that successes should make up for failures. 
Without failures progress will generally be slow because the 
approach will have been too conservative. 

The model provides for input from the providers and users 
of assistance services and for administration and coordination 
by the FESO. The reasoning behind these features is discussed 
lime Ix C. 

Specifically, the model recognizes that the investment 
required to respond to some assistance requests will not pro- 
vide a measurable, commensurate return. This is not say that 
satisfying a particular request does not provide benefit, for 
Satisfaction itself is a form of benefit. However, it is 
convenient from an operational viewpoint, and less discon- 
certing to all concerned, to classify those requests having 
marginal or intangible benefit as providing immeasurable or 
"Zero" benefit, when measuring effectiveness. 

No attempt is made to apportion credit to individual 
requests, Or among the various requesting and responding 


groups. In this way, the overall rewards for participation 
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in the program are retained without the introduction of. risk 
which would attend being involved in a request which was later 
Shown to be of marginal value. In short, there was a conscious 
effort to avoid any appearance of penalizing the participants, 
whether they are located at CEL or at shore activities. 

It is believed that this system will provide information 
NAVFAC Code 03 needs, to know whether it is doing the right 
thing in funding the Assistance Program with mission manage- 
ment funds. 

Two questions can be asked in relation to justifying the 
investment of funds in the Assistance Program. These ques- 
tions are: 


l. Did the Assistance Program produce an acceptable level 
of benefit? 


2. What level of benefit was achieved by the Assistance 
Progr am? 


The answers to both of these questions are valuable, but 
each carries a different cost in terms of effort and timeli- 
ness of availability. Efforts to determine whether the Assist- 
ance Program produced an acceptable level of benefit during 
a given fiscal year could be carried out and completed during 
that fiscal year. Results would be available to management 
in time to influence decisions concerning program efforts in 
the following fiscal years. By contrast, evaluation efforts 
to determine the overall level of benefit for a given fiscal 
year would require more effort and could not be completed 
until after the end of that fiscal year. Thus valuable man- 


agement information would not be available until after final 
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decisions had been made on funding of the next fiscal year's 
efforts. Also management pressure for this information could 
be expected to result in a period of concentrated evaluation 
effort shortly after the end of the fiscal year. 

From an operational viewpoint, there was strong intuitive 
appeal for answering the first question. Yet, it is believed 
that effectiveness information on the overall level of benefit 
would be more useful, and therefore of more interest to man- 
agement. In view of the importance of these two divergent 


views, an attempt was made to satisfy both of them. 


D. EXPANDING THE FESO AUTOMATED INFORMATION SYSTEM 

The feasibility of adapting a computer software package 
for a small service organization had already been demonstrated 
by Suess and Thaler [Ref. 9]. A computerized information sys- 
tem had been developed and demonstrated for the FESO using 
the Statistical Package for Social Sciences (SPSS). This sys- 
tem is currently being used by the FESO, a testimonial to the 
value of that effort. 

Suess and Thaler stressed the importance of adequate sup- 
porting documentation in developing the information system. 
It was considered no less important when modifying the exist- 
ing system, in that it would enable the FESO to understand 
and effectively maintain the automated information system 
using the SPSS. Accordingly, detailed documentation of the 
processing logic and operating procedures for expanding the 
FESO Automated Information System to process information on 
benefit was included as part of this study. The expansion of 


this system is discussed in Section VI. 
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Մ. FEASIBILITY OF MEASURING EFFECTIVENESS BY SAMPLING 


A. OVERVIEW 

The answer to the question of whether the Assistance Pro- 
gram produced an acceptable level of benefit during the 
fourth quarter of FY-76, when l45 requests were received, 
could be obtained by a somewhat simplistic sampling procedure 
which had considerable operation appeal. By contrast, the 
answer to the question of what level of benefit was achieved 
during that period would require application of a more rigor- 
००5 procedure. Both procedures were pursued. Either of them, 
ıf shown to produce reasonably accurate estimates of benefit, 
would provide NAVFAC Code 03 with on-going information on 
program effectiveness within available CEL resources. 

This section describes the efforts to develop and demon- 
strate an operational system to obtain adequate information 
on effectiveness measures with an affordable level of effort. 
Because of its detailed nature, the results obtained from the 
various approaches are previewed in this subsection, with the 
details included in subsections to follow. 

The original approach was limited to determining whether 
the Assistance Program had produced an acceptable level of 
benefit. The initial procedure was to subjectively rank each 
request in terms of ranges of estimated potential benefit. 
Then the evaluation of individual requests with potentially 
large benefits would proceed and continue until the aggregated 


benefit exceeded the total cost of providing assistance. At 
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that point evaluation efforts would stop. Although these 
procedures were not followed exactly, in that efforts were 
not concentrated on evaluating potentially large benefits, 
it was shown that the benefits of ten evaluated requests, or 
approximately seven percent of the total of ]45 requests, 
exceed the $53,766 cost of the program. 

The direction of the study was then changed to an approach 
which measured effectiveness in terms of ROI, based on the 
evaluation of requests in randomly sampled groups of requests. 
The sampling population was reduced to ll5 requests by elim- 
inating 25 requests subjectively ranked as producing "zero" 
benefit and five duplicate requests. Three trials were made 
using a multiple group sampling procedure with three, four, 
and then five groups of four requests each. Ranges of total 
estimated benefit were then calculated assuming a 95% confi- 
dence interval. They were too wide. In addition, the lower 
limit was negative and therefore unacceptable. The range 
narrowed and converged toward acceptable limits with each suc- 
ceeding trial. This was caused by the small number of sample 
groups and the wide range of individual benefits. This samp- 
ling procedure was abandoned before the lower limit of the 
range of estimated total benefit assumed a positive value, 
although the benefits of 20 requests, or 37 percent of the 
total requests, had been included in the calculations. 

The random sampling approach was then changed to include 
a procedure that parallels attribute inspection in the manu- 


ԱՐԻԱ व 5 09, Six additional trials were made using a 
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grouped sequential sampling procedure. The group size was 
mmegaessively increased to include 4, 8, l2,#l5, l8, and then 
45 requests. Ranges of estimated total benefit were again 
produced, which were consistently narrower than those produced 
by multiple group sampling. An acceptable range was achieved 
with the second additional trial (No. 5) which included eight 
requests, or seven percent of the total. The range was still 
quite wide, but narrowed with each succeeding trial. Results 
based on the benefits of 45 evaluated requests indicated that 
the minimum acceptable estimated ROI for the Assistance Pro- 
gram during the fourth quarter of FY-76 would be realized 
within a range between seven and 24, with a confidence of 95 
percent. 

In view of the concern for operational simplicity, the 
potential for expanding the original subjective ranking approach 
to produce estimates of total benefit was then explored. Using 
this approach, the total benefit was estimated to be $707,027,) 
which corresponded to a return on investment of 23. 

It was also shown that the Assistance Program had produced 
an acceptable level of benefit. Identified benefits of the 45 
evaluated requests totaled $324,027, or a return on investment 


of six. 


B. SELECTION OF EVALUATION TECHNIQUES 

Before the study had progressed vary far, the decision was 
made to have the EFD RDT&E liaison engineers evaluate the 
requests from their organizations as had been done in the FY- 


72, FY-73, and FY-74 surveys. They were requested to 
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coordinate the evaluation of those requests with the author 
administering the evaluation of requests from other organiza- 
tions. A modified questionnaire was developed and used to 
obtain evaluation data from field users. The development of 


this questionnaire is discussed in Appendix D. 


C. SUBJECTIVE RANKING 

A review of the data from the three previous surveys 
revealed a wide range of individual benefit values with a 
small number, about five percent, of relatively large benefit 
values included among them. This review provided the basis 
for the original approach, which was to determine only if the 
Assistance Program had produced an acceptable level of bene- 
fit, i.e., a return on investment of l.0. Some other minimum 
value of ROI could be selected, for this value defines what 
is considered an acceptable level of benefit, but recovery 
of cost was considered to be a minimum requirement. 

Each of the 745 requests were subjectively ranked in terms 
of their estimated potential benefit using the same benefit 
value ranges included in the revised form for recording verbal 
requests for assistance. (Appendix E.) The results of this 
ranking effort, to subjectively identify the "larger" benefits, 
indicated that a total of five requests were each estimated 
to provide potential benefit in excess of $25,000 (Table I). 

The operational appeal of this approach was increased with 
the thought that evaluating those five requests, or less than 


four percent of the total requests, might identify benefits 
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faa: amounted to more than twice the cost ($53,766) of pro- 
viding assistance services during the fourth quarter of FY-76. 

At the time the evaluation of those five requests was 
ready to begin, the expanded FESO Automated Information System 
was ready to be tested with live data. Rather than wait to 
obtain data from evaluating the "larger" benefit requests, 

Six requests which had been evaluated by the EFD RDT&E liaison 
engineers were chosen so as to provide a variety of data for 
test purposes. The total benefit from these six requests was 
determined to be $46,828, or 87 percent of the cost of the 
Assistance Program. 

As testing of the expanded information system continued, 
and again without concentrating on the "larger" benefit 
requests, the number of evaluated requests included in the 
data base was increased from six to ten. Three more requests 
evaluated by the EFD representatives were added, and one 
request evaluated by the author. These four included one 
request which had been evaluated as a duplicate, and therefore 
of zero benefit. The total benefit of these ten requests 
amounted to $54,428. Thus identified benefits from approxi- 
mately seven percent of the total requests, had exceeded the 
cost of the program. In addition, the identified benefit of 
nine of these requests was in the range of potential benefit 
as ranked by the author. The results of these ten evaluations 
are shown in Table II. 

Although the results were encouraging, serious doubts 
tore forming as to thelr credibility and usefulness to man- 


agement. There would be some who would question the approach. 
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TABLE I 


SUBJECTIVE RANKING OF ASSISTANCE REQUESTS 


Benefit Estimate Range of Estimated Number of Percentage 





Code EE ra MBenefit (F) Requests ef Total 

l 0 25 7 
2 Օ - 499 l9 T3 
3 500 - 4999 73 50 
4 5000 - 24999 ]8 l 2 
5 Over - 25000 5 4 
6 (Duplicate) 0 5 4 

TOTAL 745 l00 

TABLE Il 


BENEFITS OF FIRST TEN ASSISTANCE REQUESTS 


Benefit Estimate 


Request Number Code Benefit (S$) 

BISE Six 557 l 0 
643 4 237862 

647 J: 0 

682 á 20,000 

688 3 pel © 

720 3 2 25 

Next Four >33 5 5,000 
587 6 0 

588 5 0 

TS 3 3,600 

TOTAL 54,428 
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Gaererore, the direction of the study was modified and the 
scope expanded to overcome potential barriers to future use. 
The direction was changed to determine what level of bene- 


fit had been achieved, stated in terms of ROI. 


D. RANDOM SAMPLING APPROACH 

In this approach, requests were randomly selected, evalu- 
ated, and their individual benefits aggregated and used as a 
basis for estimating the total benefit of all requests 
received. The estimated total benefit was then divided by 
the total investment in the Assistance Program to determine 
an estimated ROI. The aim was to achieve significant results 
with the minimum size sample, so that evaluation efforts in 
an on-going system could be maintained at a manageable level. 
Somewhat arbitrarily, although based on results to that point 
in the study, a ten percent sample size was selected. With 
a sampling population of ll5 requests, the sample consisted 
or l2. 

A multiple group sampling procedure using three separate 
sample groups of four requests each, with replacement taking 
place before each new group was selected, was chosen as the 
most appropriate procedure for this sample size. The premise 
was that the mean and standard deviation of the sample group 
means would provide the best estimate of the mean of the 
population of requests sampled. 

The SPSS provides a procedure for taking a random sample 
from a file (population) for processing. A factor is selected 


which is a positive number less than l.0 which indicates the 
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pereentage of the file that is to be analyzed. For instance, 
a factor of 0.l would result in approximately a lO percent 
sample for the entire file. The probability of selecting any 
particular request in the file is equal to the factor speci- 
fied. However, as each request is considered for selection 
independently of all other requests, the resulting set of 
sampled requests will generally not be exactly the size speci- 
fied. The sample procedure used by SPSS is set up in such a 
way that a different random sample will be selected each time 
the user requests a sample [Ref. l0]. 

A factor of 0.033 was selected to obtain the three groups 
of four requests each for a series of multiple sampling. 
However, rather than obtaining this grouping, the SPSS proce- 
dure consistently provided samples containing three, two, and 
five requests. Repeated attempts to obtain three or more 
equal size samples proved fruitless and efforts to use the 
SPSS sample procedure were abandoned. 

Using the random number generator program in an HP-25 
programmable nand-calculator, three groups of four requests 
each were selected. Field evaluations via telephone inter- 
views were performed by the author for those selected 
requests not already evaluated by the EFD RDT&E liaison 
engineers. The resulting mean of the mean benefit of these 
three sample groups was $l2,l34, with a standard deviation 
of $9,607. Although the benefit values of individual 
requests are assumed to be normally distributed, the sampling 


distribution was probably not normal since the sample size 
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was small (less than 30). Accordingly, the student's "t" 
distribution for small sized samples was used [Ref. ll]. The 
95% confidence interval of the mean benefit was determined to 
range between -$Sll,734 and $36,002. The corresponding range 
of total benefit for the population of ll5 requests was from 
-22/349,40 ऋ0 Տ54,40,230. Thus the estimated range of ROI 
was between -25 and 77 with a confidence of 953. 

A review of these figures in the light of the benefits 
already identified spoke for the need to seek a narrower 
confidence interval. 

Another set of requests was randomly selected to comprise 
the fourth sample group. Again, missing evaluations were ob- 
tained from telephone interviews. The sample then consisted 
of l6 requests, divided into four groups. The resulting mean 
of the mean benefit shifted to the left and the confidence 
interval was narrowed somewhat, although it was still not 
definitive since the lower limit remained negative. The 
resultant range of ROI was -ll to 52. 

The number of sample groups was increased again, to five, 
in an attempt to gain further narrowing. The process of ran- 
domly selecting another group of four requests, obtaining 
missing evaluations and performing the necessary calculations 
was repeated as above. The results were encouraging in that 
the range of ROI did narrow further, although the left end 
remained negative. The results of these three sampling trials 


are shown in Table III. 
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At this point it was tentatively concluded that the diffi- 
culty in obtaining a fairly narrow and definitive range of 
estimated total benefit was mainly due to the small number 
of sample groups and the wide range in value of individual 
benefits. It was believed that these complications would not 
be present, or at least not to the same degree, in on-going 
evaluations with a larger sample population. 

A similar situation, in which a wide range of individual 
benefits is fixed, but the sample size can be effectively 
increased, exists in the manufacturing industry, where inspec- 
tion of entire lots of goods is too expensive. Sampling 
inspection by attributes is used with the intent of achieving 
the desired type of control information with the smallest 
possible sample. In an attempt to reduce the total amount 
of inspection required, sequential sampling is used. This 
procedure consists of inspecting one item at a time, or one 
group of items at a time, and after each inspection deter- 
mining whether the accumulated information from items thus 
far inspected justifies a decision concerning acceptance or 
rejection of the lot, or whether another item or group of 
items must be inspected. A judgement concerning the whole 
lot is made on the basis of the combined samples [Refs. l2 
II). In attribute inspection applied to manufacturing, 
upper and lower acceptance limits are established for defec- 
tive items as the basis for an accept or reject decision. 

In this study, where the attributes are benefits, an "“accept- 


able" range was achieved when the upper and lower confidence 
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limits were both positive. This amounts to rejecting the 
Situation where a large isolated benefit or a significant 
number of zero benefits produce non-definitive results. An 
acceptable range may still be fairly wide, although evalua- 
tion of additional groups will narrow the range further. 

Grouped sequential sampling was chosen so that results 
could be compared with those previously obtained from multiple 
group sampling. Specifically, a fourth trial was run with 
the evaluated benefits of the four requests in the first 
group of the multiple group sampling procedure used to cal- 
culate estimated total benefit. If the sample were suffici- 
ently convincing, a decision could be made about the entire 
program at that point. Previous results did not indicate 
that this would be so, although it might be in future evalu- 
ations. Accordingly a fifth and sixth trial group were 
tested using combined samples of eight and l2 requests. At 
that point duplicate requests appeared in the random sample 
groups which had the effect of reducing the combined samples 
for the seventh and eighth trials to l5 and ]l8 requests. 

One more possible test of this procedure existed. A total 
of 45 requests had been evaluated at that time by the EFD 
RDT&E liaison engineers and the author. These did not include 
any duplicate requests or those ranked in the "zero" benefit 
range, which were excluded from the random sampling approach. 
աՆ 5 consisted of testing these 45 requests as 
one sample to see whether a narrower range of estimated total 


benefit would result. This was a biased trial, although the 
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inaccuracy was believed to be minimal since 40 percent of the 
requests had been randomly selected. Also, since the sample 

of 45 requests represented 39 percent of the total of 2)5 
requests in the population, it should produce a fairly reli- 
able estimate of total benefit. The results of the six grouped 
sequential sampling trials are shown in Table IV. 

The identified benefits of the 28 requests in Trial No. 8 
amounted to $245,844 97१6 the identified benefits of the 45 
requests in Trial No. 9 totaled $324,029. ११€३९८ values pro- 
vided sufficient evidence to conclude that it was feasible 
to obtain meaningful ranges of estimated total benefit with a 
random sampling approach, and no further attempts were made 
to narrow the range. At that point it had been determined 
that the estimated ROI for the Assistance Program, during the 
fourth quarter of FY-76, would fall within a range between 
seven and 24, with a confidence of 95 percent. 

Several significant differences were apparent from a com- 
parison of the results from grouped sequential sampling with 
those from multiple group sampling: 

l. The range of estimated total benefit was wider with 
multiple group sampling, and although it narrowed faster and 
more uniformly than with grouped sequential sampling, the 
lower limit remained negative after benefits of l6 percent 
of the requests had been included in the samples. 

2. The range of estimated total benefit was consistently 
narrower with grouped sequential sampling, and the lower limit 


became vositive by the time benefits of seven percent of the 
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TABLE IV 


ESTIMATED TOTAL BENEFIT BY GROUPED SEQUENTIAL SAMPLING 
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TABLE IV (Continued) 


ESTIMATED TOTAL BENEFIT BY GROUPED SEQUENTIAL SAMPLING 
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requests had been included in the sample group, although the 
range was still quite wide. 

The results of the nine random sampling trials are shown 
graphically in Figure 4 for comparative purposes. Since the 
calculations of statistics were performed external to SPSS, 
using an HP-25 programmable calculator, the procedures were 
not included in the modified FESO Automated Information Sys- 


tem. However, typical calculations are included in Appendix F. 


E. ESTIMATING APPROACH 

With the knowledge that a random sampling approach would 
produce meaningful benefit information for management use, the 
possible value of subjectively ranking requests into ranges 
of estimated potential benefit was explored further. By 
multiplying the number of requests in each range by the mid- 
value of that range, and summing these range totals, an esti- 
Mate of total benefit could be obtained. Since the mid-value 
of the "over $25,000" range was unknown, the value of $25,000 
was arbitrarily assumed. Performing these calculations 
resulted in an estimated total benefit of $600,500 (Table V). 
The value was well situated within the 95 percent confidence 
irc ‘Of from S35l,440 to $Sl,304,560 obtained in sampling 
աաա NO 9. Actually, it was conservative as it was 38 per- 
cent below the mid-point value of $828,000. 

An opportunity existed to see how conservative the benefit 
estimating approach was, by comparing it with the identified 
benefits of the 45 requests included in Trial No. 9. Using 


those 45 requests, the procedure described above was repeated. 
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The estimated total benefit was calculated to be $263,508, 
compared to the identified total benefit of $324,0l7. The 
estimating approach was conservative by l9 percent. 

A review of the individual benefits of the 45 requests 
showed that three requests produced benefits greater than 
$25,000. The values of these benefits were $75,l65, $39,770, 
and $30,000. Their mean value was $48,3l2. Using $48,000 
for the mid-value of the "over $25,000" range should increase 
the reliability of the estimate since it would then be based 
on actual benefit data. Recalculating the total benefit 
using this revised mid-value resulted in an estimated value 
of $7l5,500. This value was closer to the mid-point value of 
$828,000 from Trial No. 9, but still l4 percent below it. 

Using $48,000 for the mid-value of the "largest" range 
and repeating the comparison with the 45 evaluated requests in 
Trial No. 9 resuited in an estimated total benefit of $332,500. 
A difference of only three percent from the identified total 
benefit of $324,0l7 was accepted as an indication that the 
estimating approach, as practiced by the author, had been suc- 
cessfully demonstrated to be reasonably accurate. 

After that successful demonstration, an estimate of total 
benefit was made based on the actual benefit that had been 
identified and the estimated benefit remaining in the 70 
requests that had not been evaluated. The results of these 
calculations indicated an estimated total benefit of $707,0l7 
(Table VI). Although this value was very close to the origi- 


nal estimated value of $7l5,500, it was not taken as an 
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TABLE V 


ORIGINAL ESTIMATE OF TOTAL BENEFIT 


Range of Estimated Mid-value Number of Estimated 
Potential Benefit ($) of Range ($) Requests Benefit ($) 

0 0 30 0 

0 - 499 250 l9 4750 

500 - 4999 2750 iS 200750 

5000 - 24999 ]5000 L3 270000 

over 25000 25000 5 १०5000 

TOTAL 45 600500 

TABLE VI 


ESTIMATE OF TOTAL BENEFIT BASED ON EVALUATED BENEFIT 








Range of Estimated Mid-value Number of Estimated 
Beeeneral Benefit ($) of range ` Requests Benefit (S$) 
0 0 30 0 
0 - 499 250 9 2250 
500 - 4999 2750 49 EFS 
5000 - 24999 5000 J) ]50000 
over 25000 48000 2 96000 
SUBTOTALS 70 383000 
EVALUATED REQUESTS 45 324027 
TO LAS LS 707.027 


48 





indication that these estimates of total benefit were any 
better than those obtained by the grouped sequential sampling 
procedure. What was apparent, was that the estimating approach 
provided another valuable tool for the FESO to use in estim- 
ating total benefit with reasonably accuracy. For this rea- 
son, procedures were incorporated into the expanded FESO 
Automated Information System to facilitate these calculations. 
These procedures, along with other system modifications are 


discussed in section VI. 
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VI. EXPANSION OF THE FESO AUTOMATED INFORMATION SYSTEM 


Using the Statistical Package for the Social Sciences 
(SPSS), an automated information system was developed and 
demonstrated by Suess and Thaler [Ref. 9]. It has since been 
implemented to satisfy the data processing requirements of 
the FESO using the Lawrence Berkeley Computing Center via a 
batch terminal. 

Since the gathering and processing of information on the 
benefits of assistance requests will be an on-going require- 
ment for the FESO, it was a logical next step to accommodate 
by expanding the FESO Automated Information System. The 
expansion of the system to satisfy the information require- 
ments of an effectiveness measuring system is discussed in 


eneee nder of this section. 


EENI OF SYSTEM OBJECTIVES 
The major objectives in expanding the FESO Automated 
Information System are: 


l. To relieve the FESO of as much time-consuming manual 
data manipulation as possible. 


2. To provide a method to process benefit information on 
any or all requests received during any or all quarters of 
a fiscal year. 


3. To provide a method to verify automatically the 
accuracy of the data input to the computer. 


4. To accurately process and generate all of the benefit 


information required by the FESO in preparing effective- 
ness reports. 
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Be IDENTIFY OUTPUT REPORTS 

The initial step in expanding the system was to identify 
required output reports. To ensure adequate identification 
for control purposes, each output report was given a descrip- 
tive title and assigned a report number. 

Previous efforts had identified eleven reports as being 
necessary to provide the FESO with the information to function 
effectively. Eleven additional reports relating to benefit 
information were identified as being necessary and were given 
individual designators. An inventory of these reports is in- 


cluded in Figure 5. 


eee PARE REPORT SPECIFICATIONS 

After the output reports had been identified, input and 
data manipulation required to produce the output reports were 
determined. This was done by preparing a Report Specification 
for each output report listed in the Inventory of Reports. 
Preparation of the Report Specifications required a detailed 
systematic analysis of each output report to determine the 
data selection criteria, data elements, data manipulation 
logic and output varameters. The report specifications for 
the eleven benefit reports are included in Appendix G. The 
information contained in the report specifications of the 
benefit portion of the information system was used to prepare 
the SPSS control cards and operating procedures described 


later. 


Sa 





Report No. 
OB 
l2 
Və 
l4 
J5 


MG 
ML 
l8 


१82. 


INVENTORY OF BENEFIT REPORTS 


Benefit Edit Report 

Original Estimate of Total Benefit 
Breakdown of Evaluation Status 
Breakdown of Field Evaluation Data 


Comparison of Contribution and Information 
Provided 


Summary of Savings by Type of Savings 

Estimated Minimum Benefit of Evaluated Requests 
Summary of Evaluated Benefits by Type of Savings 
Statistics for Evaluated Benefits 

Summary of Benefits by Benefit Code 


Estimated Minimum Remaining Benefit by 
Benefit Estimate 


Comparison of Benefit Estimate and Benefit 


Figure 5. Inventory of Benefit Reports 
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D. COMPILE DATA ELEMENT LIST AND ASSIGN CODES 

With the report specifications prepared, the requirements 
for the data input became apparent. A review of the report 
Specifications provided a composite list of all data elements 
that should be included as input to the preparation of the 
reports. A list of all data elements used in the benefit 


portion of the information system is shown in Figure 6. 


E. DESIGN INPUT RECORD FORMAT AND CODE DATA DEFINITION 

The structuring of the data elements followed from the 
format of the questionnaire (input record). Figure 7 shows 
the format of the input record for assistance benefit inform- 
ation. Fields l through 4 comprise the control group and are 
identical to those previously used, as in Field 5. Field 5 
is a special type field. It is a redefinition or combination 
of fields 3 and 4. Fixed data comprise Fields 6 through 25 
and 20 through 22. Fields 26 through l9 and 23 comprise the 
variable group. 

With the input record format described, the additional 
data base was described in computer language utilizing the 
data definition language feature of the SPSS. This process 
entailed defining the data elements and their codes in the 
format specified in the SPSS Manual. The results of the prep- 
aration of data definition cards describing the expanded 
portion of the Assistance Program data base are listed in 
Appendix H. Cards which relate to the benefit portion of 
the data base, or required modifications to accommodate it, 


are preceded by a solid ball (ə). 
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Data Element 


Evaluation 


LIST 


DATA ELEMENT 


Estimate of Benefit 


Any Benefit 

No Benefit 

Benefit Code 

Use of Advice or Assistance 

Use Factor 

Discounted Present Value Factor 
Type of Savings 


One-Time Construction or Repair 
Cost Savings 


Annual Maintenance Cost Savings 


Annual Operating Cost Savings 


Project Cost 


Sentribution of Advice or 


Assistance to Decision 


Percent of Information Provided 


by CEL Compared to Total 


Would Solution be Applicable to 
Other Activities 


Benefit 


Figure 6. 


Number of 


Characters 


i 


i 


ԻԺ = 


Data Element List 
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Format 

Numeric 
Numeric 
Numeric 
Numeric 
Numeric 
Numeric 
Numeric 
Numeric 
Numeric 


Numeric 


Numeric 
Numeric 
Numeric 


Numeric 


Numeric 


Numeric 


Numeric 
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F. CONVERT REPORT SPECIFICATIONS INTO COMPUTER TASK 
Per INTTIONS 


A set of task-definition cards was then prepared to 
describe the data manipulation and the statistical calcula- 
tions to be performed on selected data for each report to be 
produced. The data-definition cards remain unchanged while 
preparing all reports. However, a set of task-definition 
cards is required for each report. Each set of cards was 
identified by using a TASK NAME statement card. 

The report specification provided the information neces- 
sary to prepare the task-definition cards. The data selection 
constraints defined which data should be used. The data 
manipulation requirements defined which calculations were to 
be made. The task-definition logic for each of the benefit 
information reports developed for the Assistance Program are 


contained in Appendix I. 


G. DEVELOP EDIT LOGIC 

In developing the editing logic for the benefit portion 
of the Assistance Program data base, "Sight" verification of 
the input data was implemented through the use of the activity 
file, a computer printout of all input data. 

Prior to the processing of any reports, the input data 
was run through an edit operation. This edit operation was 
designed by the preparation of an additional set of task- 
definition statements. This set of task-definition statements 
checks each data record for accuracy in conformance with the 


coded data previously established on the data-definition 
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cards in the computer program. Figure 8 lists the set of 
task-definition statements prepared for the edit operation. 
The edit logic checks each field in the input record for 
accurate and complete data in two manners. First, the con- 
tent of each field is checked against the data-definition 
section statements that defined the structured data base and 
accepted values. Each field must have an acceptable value 

Or it is flagged as an error. Next the relationship between 
fields is defined and the content of the related fields must 
be consistent. For example, if for a particular request, the 
type of saving field has been coded "l," indicating that a 
one-time construction or repair savings has been realized, 
and there is no data in the one-time savings field, the edit 
logic will flag an error. If the record passes the edit with- 
out any error flags, the data are considered accurate and 
complete. 

Error information about the data record that failed the 
edit test is displayed through the use of an exception report. 
Error flags showing the input card columns containing erron- 
eous data are displayed in the report along with the necessary 
record identification information. When the erroneous data 
card records are corrected and all input data passes the edit 
witha "Negative" exception report (no errors), processing of 
the reports may begin. 

Artificial test data were contrived for the benefit portion 
of the information system to test the edit logic. Invalid 


data were constructed to test the comprehensiveness of the 
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edit logic. After successfully passing this test, representa- 
tive live data were prepared and used as input to test each 
report separately. After each report processing logic was 
tested separately, a system test was conducted using live 
data and generating all benefit reports. 

The test reports prepared for the benefit portion of the 
Assistance Program data base provided some interesting results 
which were discussed in section V. To illustrate a typical 
test report, the test report used as a basis for Table II is 
shown in Figure 9. The reason that Table II lists ten 
requests, while Figure 9 lists only seven, is that three of 
the requests were ranked as "zero" benefit requests which ex- 


cluded them from field evaluations. 
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VII. CONCLUSIONS AND RECOMMENDATIONS 


a. CONCLUSIONS 

The arguments for and the demonstration of the use of two 
conceptually different approaches to satisfy NAVFAC manage- 
ment requirements to measure the effectiveness of a el 
service program have been presented in this study. The follow- 
ing conclusions have been drawn. 

l. The data resulting from user evaluations of randomly 
selected assistance requests, when processed by a computer- 
aided information system is a viable and economically afford- 
able basis for estimating the return on investment of the 
resources expended on the Assistance Progran. 

2. Grouped sequential sampling procedures produced mean- 
ingful ranges of estimated total benefit for requests received 
during the fourth quarter of FY-76. The total benefit for 
that period was estimated to be between $350,000 and $4, 300, 000, 
which corresponded to a return on investment between seven and 
De with a confidence of 95 percent. 

3. The Assistance Program produced an acceptable level 
of benefit during the fourth quarter of FY-76. Benefits quan- 
tified in this study amounted to $324,0l7, or a return on 
investment of six. 

4. An estimating approach based on subjectively ranking 
requests into ranges of potential benefit is a valuable tool 
for the FESO to use in estimating total benefit of the Assist- 


ance Program with reasonable accuracy. Using this approach, 
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the total benefit of the Assistance Program during the fourth 
quarter of FY-76 was estimated to € $720,000, or a return on 


investment of ]3. 


B. RECOMMENDATIONS 

MEQ requese should be subjectively ranked by the 
FESO into one of two categories; those containing "zero" 
benefit, and those with potentially measurable benefit. "Zero" 
benefit requests should be excluded from further evaluation. 

2. An effectiveness measuring system, using a random 
sampling approach as presented in this study, should be 
adopted by the FESO for measuring the effectiveness of the 
Assistance Program during FY-77. 

3. Evaluation data for selected requests should be ob- 
tained from field users, using the feedback questionnaire 
developed as part of this study. Requests originating from 
the EFDs should be administered by the EFD RDTSE liaison 
engineers. The FESO should administer the evaluation of all 
other selected requests. 

4. Four groups of four requests each should be selected 
from each of the four quarters of FY-77, which should approxi- 
mate a ten percent sample. Evaluation data from these 
requests should be used to calculate estimated total benefit 
and return on investment using procedures for multiple group 
sampling, grouped sequential sampling, and then request~-by- 
request sequential sampling as the basis for selecting a single 
sampling plan and an appropriate sample size for on-going 


effectiveness measurements. 
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5. Additional work should be undertaken by the FESO to 
locate or develop procedures for selecting sample groups and 
programs to perform the necessary statistical calculations 
for the selected sampling procedure which are compatible with 
the SPSS, or can be incorporated into the FESO Automated 


Information System. 
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APPENDIX A 


CURRENT PERFORMANCE MEASUREMENTS 


In l97l, the FESO initiated measurements to gauge the 
activity of the Assistance Program and its usage by field 
personnel. The Assistance Program is currently measured by 
parameters, which are indicators of activity, rather than 
effectiveness. For the most part, they measure level of 


effort or input, rather than output. 


A. NUMBER OF REQUESTS 

The number of requests measurement is simply a count of 
the individual assistance requests received from personnel 
located at shore activities. This is one of the two primary 
measures used to gauge usage of the Assistance Program. A 222 
percent increase in usage over the six years that measurements 
have been made is encouraging. The results of these measure- 
ments are shown in Figure l0. During the fourth quarter of 


FY-76, l45 requests were received from shore activities. 


B NUMBER OF ACTIVITIES 

The number of activities measurement is a count of the 
individual activities whose personnel have requested assist- 
ance. This is the other primary measure for evaluating usage 
of the program. These two measure were selected and are used 
together, because it is felt that it is better, in terms of 
value to the Navy, to receive one request from each of ten 


activities, than ten requests from one activity. The magnitude 
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of the number of activities count is smaller than the number 

of requests, simply because there are more people than there 

are shore activities. A l55 percent increase in usage, over 

the same siX-year measuring period, is also encouraging. The 
results of these measurements are shown in Figure ll. During 
the fourth quarter of FY-76, requests were received from 


personnel at 78 individual shore activities. 


ER SOWRECE OF REQUESTS 

The source of requests measurement categorizes the number 
of requests received by the type of organization that the 
individual requestor represents. The source of requests by 
type of organization for requests received during the fourth 


quarter of FY-76 are shown in Figure l2. 


र THOD OF REQUEST 

The method of receipt categorizes the method by which 
the individual requests were initially received by CEL 
personnel. Information on the method of receipt for requests 
received during the fourth quarter of FY-76 is shown in 
Figure l3. The fact that 88 percent of the requests were 
initially received by telephone is an encouraging acceptance 
of CEL's philosophy that it should be easily reached by 
telephone. 
E. SUBJECT OF REQUESTS 

The subject of requests measurement is a categorization 
of the number of requests received into broad subject cate- 


gories. This measurement is expressed as a percentage and 
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Pn tO illustrate that the full breadth of CEL's capabili- 
aes are employed in providing these services to shore acti- 
vities. The subjects of requests received during the fourth 
quarter FY-76 are shown in Figure l4. 

FE RESPONSE TIME 

The response tim measurement is simply a count of the 
calendar days that elapse after receipt of a request by CEL, 
until it is satisfied. This measure is the only one that 
approaches a measurement of output--timeliness of response. 
The results of this measurement for requests received during 
the fourth quarter FY-76 are shown in Figure l5. This is a 
display of the elapsed time to satisfy a request by the number 
of requests satisfied within that period of time. The fact 
that 72 percent of the requests received. during the fourth 
quarter were satisfied within seven calendar days speaks well 
of the capability of CEL personnel to respond to field needs 
in a timely manner. 

These measurements are useful in that they indicate an 
active and growing program, but with one exception they suffer 
from the disadvantage of measuring effort. Additional param- 
eters, that provide meaningful measures of output and benefit 
are now required. 

One current shortcoming in the administration of the 
Assistance Program is the lack of organized feedback from its 
users. An effectiveness measuring system will provide a lim- 


ited amount of feedback information, but on a continuing basis. 
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This information will be useful in guiding and improving 
the program and expanding the potential for multiple use of 


certain successful applications of research results. 
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APPENDIX B 


ADDITIONAL BENEFITS OF THE ASSISTANCE PROGRAM 


The benefits of the Assistance Program are many and varied, 


and personnel at CEL and shore activities both share in these 


benefits. Since this study dealt only with those benefits 


that could be measured to some quantitative degree, it was 


considered important to list them so that they will not be over- 


looked and forgotten. Some of the additional benefits of the 


Assistance Program are that it: 


Promotes two-way communication and hence better under- 
standing between laboratory and field personnel. 


Accelerates the diffusion and adoption of new knowledge. 


Provides valuable contact for laboratory personnel with 
actual field problems. Often this is "hands-on" experi- 
ence gained when a site visit is required. 


Identifies actual field needs that would be amenable to 
solution through the R&D process. 


Provides feedback information to those involved in the 
R&D process, which serves to guide the direction of the 
research program toward efforts related to common field 
needs, the solution of which would have far-reaching 
impact. Although there was no attempt made to assess 
the value of this benefit, there is little doubt that it 
exists. Some future effort might attempt to evaluate 
the increased effectiveness of a research program as a 
result of having better knowledge of the actual needs 
of the end users of the research. For instance, an 
improvement of between one and two percent of CEL's 
program would offset the annual cost of the Assistance 
Program. 


Provides information not otherwise available to field 
personnel. 


Saves the time of field personnel who can avoid searching 


for information via literature searches and technical 
reports. 
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- Avoids the selection of some less than optimum alterna- 
tives by field personnel. 


- Provides CEL researchers with the potential for almost 
instant job satisfaction through participation in the 
immediate solution of actual field problems. 
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APPENDIX C 


CRITERIA FOR INPUT TO AND PERFORMANCE OF THE EVALUATION PROCESS 


EM IMEUT CRITERIA 

Before proceeding with the development of a system model 
to relate individual evaluations to an effectiveness deter- 
mination, Some criteria had to be established to guide the 
evaluation effort. Since the procedures attempt to measure 
benefit to users, the users must be involved, but not to the 
extent that they will view CEL, lts personnel, or the evalua- 
tion system as a threat to themselves or to their organization. 
Nor should they be made to feel that the effort is a busy, 
self-serving one on the part of CEL. 

For example, there is little value in asking a person 
questions related to the benefit of the response to a particular 
request, when an experienced decision maker could subjectively 
estimate that little if any quantifiable benefit resulted from 
the exchange of information between CEL and field personnel. 
Again, this is not to say that the exchange did not provide 
benefit. It is only a realization that quantification of some 
benefits is not yet possible. If it can be judged with rea- 
sonable certainty that an assessment of benefit in quantitative 
terms would be difficult, if not impossible, why ask the field 
recuesStor for confirmation? Such tactics could be interpreted 
acerco che requestor if he thought that each request 


to CEL for assistance would be followed by an evaluation of 
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the benefit of the response he received. It might be easier 
to forego asking for assistance. In short, requests ranked 
by the FESO as containing "zero" benefit should not be evalu- 
ated in the field. 

The procedures should also involve the laboratory engineers 
and scientists who provide the assistance, but not to the 
extent that they would view the evaluation effort as unneces- 
sary Or as requiring an inordinate amount of their time. The 
Responsibility for reporting a significant part of the total 
assistance activity, those requests which are initially received 
by telephone, rests with the research personnel. If the evalu- 
ation effort reached the point where it consumed an inordinate 
amount of time, in relation to that which was required to pro- 
vide the assistance, it could become easier to not report the 
assistance request and thereby automatically avoid any subse- 
ETE Cc Valuation effort. The research personnel represent a 
resource that makes the Assistance Program possible, They 
should be protected and relieved, to the maximum extent pos- 
sible, of any unnecessary administrative effort that detracts 


from their research tasks. 


B. PERFORMANCE CRITERIA 

The question of who should perform the evaluation of bene- 
fit of individual requests is tied strongly to a concern for 
the credibility of the results, the effort and expense required 
to perform evaluations, and the possibility of adverse effects 
on the overall program. The advantages and disadvantages of 


some of the methods considered are discussed below. 
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t Eye lueteron*by+CEL Researchers by CEL Researchers 


This method would have considerable appeal to some, 
at first, but there are several major disadvantages that 
preclude it from further serious consideration. Although it 
would be administratively easy to initiate, by issuing an 
instruction requiring evaluation of each assistance request, 
the possibility of creating adverse effects would be signifi- 
cant. Another disadvantage would be a complete lack of uni- 
formity of the results. Still another disadvantage is the 
normal lack of credibility attached to self-evaluation. 

2. Evaluation by Users 

This method could involve automatically sending an 
evaluation questionnaire to each person or activity that 
requested assistance. The credibility of individual evalua- 
tions would probably be increased, but like evaluation by CEL 
researchers, the possibility of creating adverse effects would 
be significant. Also, the results from self-administered 
questionnaires would lack uniformity. In some cases they 
would not even be returned which would result in the collection 
of partial, and possibly useless, data. Because the quality 
of credibility should be strongest for data provided by users, 
their participation should be provided for in the final sys- 


tem, but with proper safeguards to ensure adequate data 


collection. 
3 By aida teion by Contract 


This would more than likely be the most expensive 
method in terms of dollar costs, while affording the least 


amount of control over the conduct of the evaluation effort. 
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Thus the risk of losing sight of the purpose for the evalua- 
tion and generating negative attitudes in the providers and 
users Of the Assistance Program is high. Such results might 
be more believable to "outsiders," since the possible influ- 
ence of self-evaluation would be removed, however, the results 
are primarily intended for internal NAVFAC and CEL use. Uni- 
formity of results might be an advantage, but this would de- 
pend upon how the evaluation effort was staffed and conducted. 
A a ELTON TES the FESO 

Given that evaluations are to be performed, they could 
not be done by the FESO without any input from the users and 
CEL personnel. If they were, the results would be entirely 
subjective and would lack any degree of credibility. However, 
an evaluation effort coordinated and administered by the FESO 
has the advantage of decreasing the burden on the CEL research 
staff, providing a high degree of uniformity of data, and main- 
taining control over the introduction of factors that would 
result in negative attitudes being generated among users. 

The method of having the evaluation efforts adminis- 
tered and coordinated by the FESO was selected for the effec- 
tiveness measuring model. With adequate input from the users 
and providers of assistance, this method provides the control 
necessary to ensure that sufficient information is provided 
to NAVFAC management, at an affordable price, and with the 


least disturbance to perticipants in the program. 
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APPENDIX D 


DEVELOPMENT OF A MODIFIED EVALUATION PROCEDURE 


A. MODIFICATION OF QUESTIONNAIRE MEASURING TECHNIQUE 

As indicated earlier, the evaluation technique developed 
by Hendrickson and Fisher represented a breakthrough. The 
major input of data used as the basis for the analysis of the 
FY-/74 assistance efforts was obtained through the use of a 
questionnaire which was revised from those used in earlier 
studies (Figure 26). A detailed review of the procedures 
and resulting data revealed that some additional information 
was gathered external to the questionnaire which necessitated 
some initial hand calculations prior to machine processing of 
the data. It was considered desirable to modify the question- 
naire to avoid these external efforts, as discussed in the 
following section. The revised auestionnaire used in this 


stuüudy is Shown in Figure 27: 


8. OPTIMIZING THE BASIC EVALUATION MODEL 

The basic evaluation model developed by Hendrickson and 
Fisher quantifies in dollar terms the benefit of information 
provided by CEL in response to a specific request from a field 
user. The model is not subject to analytical analysis and 
proof, but the results therefrom do have intuitive appeal. 
A critical review by this author failed to reveal any short- 
जा the approach or logic, and it was decided to use 


जी न model, with slight modifications, to estimate the 
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an 


BESDEEROUECT EFFECTTVENESS QUESTIONNATRE 


Ceséription of project: 











Do you consider that the advice/assistance received was in any way 
beneficial? (If CEL contact led to another source of useful information, 
check yes and include that source in answering questions 4 through 7.) 


(l) Yes (2) No 


If no, why not? (Upon completion of this question, skip questions 4 through 
7.) (check one) 


(no reply received, 
(2) lab could not provide expertise in this area. 
(3) other. Please explain: 





Describe the degree of utilization of the advice and/or assistance. 
(check as many as are appropriate) 


(]) directly implemented. 
(2) direct implementation is planned 
3 


underway. 
(4) more extensive research on proposed solution is planned 
or underway. 


(5) advice was not applicable to particular problem but useful 


On otners. 


(6) problem solved without utilization of advice/assistance. 


If implementation of advice/assistance has been accomplished or is planned, 
estimate the savings. (answer as many as are appropriate) 


DE usin one time repair or constrdesion cost. 37 
(2) Savings in maintenance cost. $ oer time period 
(3) Savings in operating cost. Š per (time period) 

(4) Other. Explain: 


) test and evaluation of recommended solution is planned or 





Figure l6 FESO Project Effectiveness Questionnaire 
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6. Do you feel the solution to the problem could be applicable to other 
activities? (check one) 


(]}) Yes (2) No 


7. Many improvement projects have intangible benefits that are independent 
of measurable dollar benefits. If CEL/RDT&E advice and/or assistance 
was utilized, then estimate the intangible effects. (check as many as 
are appropriate) 


— 


The technology was used on more than one project. 
The advice and/or assistance provided stimulus for 
subsequent solutions to other projects. 


— — 
SO} — 
— 


(3) Reduced the subsequent severity of the problem. 
(4) Increased safety factor. 

(5) Increased morale. 

(6) Increased education/training of personnel. 

(7) Dollar savings were only benefit derived. 

(2) Other intangible benefits: 


8. where did you learn of the availability of Civil Engineering Lab's 
FESO Program for assistance on field problems? 


Name of person originating request: 
Activity Position Autovon No. 


Name of person filling out questionnaire (if different than originator): 





Activity Position Autovon No. 


Project number: 


Figure l6 FESO Project Effectiveness Questionnaire 
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CEL ASSISTANCE FEEDBACK QUESTIONNAIRE 


I. Description of request: 


L 








2. Do you consider the advice/assistance given was in any way beneficial? (If 
CEL contact led to another source of useful information, check Yes and con- 
sider that information in answering questions 4 through 8.) 


(१३) Yes (2) No 


3. Լք not, why not? (Upon completion of this question, skip questions 4 through 
8) (check one) 
` (l) no reply received. 
___(2) ao advice/assistance available from CEL. 
__ (3) advice/assistance given was incomplete/insufficient. 
_ (4) advice/assistance given didn't apply to problem/project. 
_ (5) didn't agree with advice/assistance given. 
___ (6) edvice/assistance received too late. 
__(7) solution/recommendation too expensive. 


(8) other. Please explain: 





—rruaasr a P rra a a Var r p 


¿. Describe the use of the advice/assistance provided. (check one) 


(l) directly implemented. 
(2) direct implementation planned as part of: (pick one) 
(Pi eandedupLrojece iT FI 


___ (22) budgeted project in FY 





£ Uj (OS) Slanned projecc in &Y 
(3) implementation being considered. (pick one) 
(3l) test and evaluation is underway or planned. 


(32) more extensive research on proposed solution is 
underway or planned. 


(4) general information for future use 


Pilea Se Wesco ESA A er 


(5) other. 


Figure ]7 CEL Assistance Feedback Questionnaire 
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5. If implementation of advice/assistance has been accomplished, is planned, or 
is being considered, estimate the savings. (answer as many as are 
appropriate) 


(4) savings in one time construction or repair cost. $ 





(2) savings in maintenance cost. $ per (time period) 


(3) savings in operating cost. $ per (time period) 








(4) If savings are not estimable in dollars, but were intangible (re- 
duced the severity of the problem, increased efficiency, increased 
safety, increased morale and/or living conditions, etc.) what was 
the dollar value of the project, or portion of the project, to 
which the advice/assistance applied? $ 





(5) other. Please explain: 


 —— w — — m w TA 


——R_Ñ c rr r aa rc mAsa ara armasha 








6. Describe the contribution of the advice/assistance to the overall decision. 
(check one) 


(Լ) confirmed an earlier opinion/conciusion. 


(2) helped select between previously known alternative approaches/ 
solutions. 


(3) provided information used in developing a better alternative 
apprcach/solution than was previously planned. 


(4) provided a recommendation which was a completely new (not 
previously known locally) and better alternative. 
7. Of the total information used to make the decision which resulted in the 
above savings, what portion/percentage was provided by CEL? (check one) 
l0 20 30 40 50 60 70 80 90 ३00 
8. Do you feel the approach/solution to the problem/project could be applicable 
to other Snore Activities? 


(]) Yes 


Name of person who originated request: 


RECTVIZY: Autoven No. 





Name of person filling in questionnaire (if different than originator) 


EVI Ey: autovon: 





———— A Q 


Assistance number: 


Figure I7 CEL Assistance Feedback Questionnaire 
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effectiveness of the Assistance Program. A review of the 
basic approach is included along with modifications by this 
author to provide continuity. 

The basic approach taken was to identify distinguishable 
categories into which requests could be grouped. In so group- 
ing, consideration was given to assuring that the methods of 
quantifying benefits within each group was workable within 
the time resources of both the evaluator and the persons sur- 
veyed. Also, the number of groups was limited to a meaning- 
ful number relative to the size of the survey group. Thus, 
if the requestor indicated that the information which he had 
received was in any way beneficial, the request would then be 
categorized into the group which best suited the request at 
hand. The groups ranged from easily identified benefits, 
through more subjective quantification methods to a final 


unquantifiable category. 


C. CATEGORIZATION OF REQUESTS 
Hendrickson and Fisher developed a three-way categoriza- 

tion of each beneficial request. The essence of the categor- 
iweaelonmecocess, after determination of whether or not a bene- 
fit existed, reduced to the testing of each request with three 
questions: 

irs: Categorization) Did the information provided 

take the form of a specific recommendation for solution of 

a problem, or did it merely provide information which was 

combined with information from other sources to forma 


basis on which a decision for action (or in some cases, no 
action) could be based? 


2. (Second Categorization) What was the probability that 
Whe infermatiOon provided will be acted upon? That ls, has 
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the information been implemented, or is implementation 
planned, or is implementation dependent on results of 
tests, Or is the information being evaluated to determine 
the advisability of implementation? 

3. (Third Categorization) Is the realized or expected 
benefit, if implemented, estimable in terms such as reduced 
cost or eliminated cost? 

This three-way categorization resulted in seventeen dif- 
ferent groups or "benefit codes" into which a given beneficial 
request might be categorized. The complete breakdown of the 
groups into which a request could fall based on this three- 
way categorization are shown in Figure 28. 

In addition, standard methods of quantifying the present 
value of expected future benefits for each request were derived, 
and an Ordinal ranking of the subjectivity of the benefit codes 
was achieved. 

The following subsections describe how the various sub- 
jective factors were determined to be appropriate for each 
phase of the three-way categorization process. Although 
Hendrickson and Fisher developed mean values and ranges of 


accuracy, only the mean values are discussed and used in this 


study, since it is intended as a basis for operational use. 


l. mote Categorization, Action Recommended, Versus Categorization, Action Recommended, Versus 


Information Provided 

HendricksonandFisher originally conceived that CEL 
would be given ]l00 percent credit for benefits which resulted 
from a specific action recommendation. Benefits credited to 
CEL for providing partial information on a problem were to be 
reduced by an appropriate factor representing the relative 


value of the information. This factor was expressed as a 
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CATEGORIZATION OF FY-74 ASSISTANCE REQUESTS 





FIRST SECOND THIRD BENEFIT 
CATEGORIZATION CATEGORIZATION CATEGORIZATION CODE 
Estimable l 
Action Taken | 
Not Estimable 
Estimable 4 
Action planned — 
Not Estimable մի 
Action 
Recommended Estimable 7 
Test Required 
Not Estimable T3 
Estimable 9 
Evaluation —| 
Not Estimable rS 
Estimable 2 
ACELON Taken — 
Not Estimable 5 
Estimable 6 
Action Planned 
Not Estimable js; 


Information 
Provided Estimable 8 
Test Requlred 


Not Estimable l4 
Estimable TO 
Evaluation —— 
Not Estimable l6 
General š Filed for Reference है 
Information 


Counted as a Zero Benefit 


Mo Beneficial 
Information 


Not Counted (Duplicate, etc.) 


Figure l8 Categorization of FY-74 Assistance Requests 
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percentage of the total information required to perform the 
action necessary to realize the benefit in question. However, 
the approach of giving CEL ]l00 percent credit for aM Հեա 
Recommended" requests was abandoned when they realized that on 
many requests much of the recommendation incorporated proce- 
dures which were well known at the field level. 

aney correctly realized that to credit CEL with the 


total benefit resulting from such a request would be clearly 


inappropriate. Thus an "information-%-factor" was applied to 
every request. The information-%-factor was assigned to each 
case by them on an individual basis. The total information 


available on the request was reviewed by Hendrickson and 
Fisher, who had first-hand experience in Shore Facilities Man- 
agement. The factor was selected from the range of .0l to २.00 
based on the relative effects the CEL-provided information had 
on the selection of the beneficial alternative available. 
Other considerations were the availability of the information 
from other sources and the relative benefit of the next best 
alternative that would have been selected had the CEL-provided 
information not been available. They recognized that the 
accuracy of subjectively assigning a relative percentage was 
less for requests where information was provided then for 
requests where a specific action recommendation was made, but 
maintained this categorization for purposes of ranking 
SOSNECCEIvVIity. 

Sm e ranking subjectivity would serve no useful pur- 


pose in an on-going system, and an information-%-factor was to 
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be applied to every beneficial request, there was no need for 
the first categorization in the modified model. Therefore, it 
was deleted. 

It was decided to gather data on the value of the 
information provided by CEL direct from the user. In this 
way, the subjectivity of the procedure would be decreased. 
This data was obtained via questions 6 and 7 on the revised 
questionnaire (Figure l7). Since question 7 required a num- 
eric answer, these answers were used in calculating individual 
benefit. It was considered that at some later date sufficient 
data would be available to determine the correlation between 
the answers to questions 6 and 7 and thereby develop some 
reliable factors to apply to the answers to question 6. This 
would allow the potential obstacle of missing data to be over- 
come should some respondent recognize the similarity of the 
two questions and elect to answer only question, but not 
question 7. 

Elimination of the first categorization simplified 
the evaluation model considerably in that half of the first 
Sixteen benefit codes were deleted. This resulted in an 
initial structure containing nine benefit codes, including 
the "General Information" benefit code. 

2. Second Categorization, Probability of Implementation 

Requests for which the response had been considered 
beneficial were classified by Hendrickson and Fisher into one 
of the following categories which were adopted "as is" for use 


in this study and are listed in descending order according to 
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the probability that the information provided would be 


implemented: 
l. Information had been implemented. (Action Taken) 
2. Implementation is planned. (Action Planned) 


3. Testing is planned or underway; implementation is 
intended if tests are successful. (Test Required) 


4. Information is being evaluated; implementation is 
intended provided evaluation indicates that expected 
benefits are probable; testing may be necessary. (Evaluation) 
5. The information provided was of a general type; not 
related to a specific project. (General Information, Filed 
for Reference) 

Based on their experience, Hendrickson and Fisher 
assigned probabilities to each of the middle three categories 
(2, 3, and 4 above) and that factor was applied to the benefit 
which would be expected if the project were implemented. The 
mest category required no factor (or a factor of l.0) since 
implementation had already been accomplished. The fifth cate- 
gory was left unquantified. 

The probability value for the middle three implementa- 
tion factors: Planning factor, Testing factor, and Evaluation 
factor, were selected and applied to all requests as appropriate. 


The mean values thev agreed on for the probability of these 


implementation factors were as follows: 


FACTOR PROBABILITY 
Planning 025 
Testing Om 
Evaluation 082 


A review of these mean probability values in the light 


of the experience of the author resulted in no reason to use 
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other values, except in the case of the Planning factor. There 
are several identifiable phases in the planning cycle which 
should reasonably be expected to affect the ultimate fate of 

a project in terms of whether it is realized or not. These 
phases are: 


J. Funded projects for which funds have been earmarked, 
although the project has not yet been implemented. 


2. Budgeted projects which are included in the budget 
requests sent forward to higher authority although not 
yet acted upon. 

3. Planned projects which are recognized as requirements, 
but whose importance is not such that they are included in 
current budget requests. 

The probability of implementation is different for 
projects in each of these phases. There appeared to be little 
difference between a Funded project that is implemented and 
one that isn't, in terms of whether or not a decision will be 
made to use a particular piece of information. The timing of 
the decision is the variable, rather than the probability that 
the project will be funded. Accordingly, a planning factor 
of ].00 was used for those projects that were funded, but not 
yet implemented. 

At the other end of the scale, no plausible reason 
could be found for using an implementation factor other than 
0.50 for Planned projects. 

For Budgeted projects, there is a greater probability 
that they will be funded compared to Planned projects, and a 
lesser probability that they will be funded compared to Funded 


projects. Thus the probability of being funded is between 


that of Funded and Planned projects. Until experience with 
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the system indicates that the use of another factor would be 
More appropriate, a factor of 0.75, i.e., midway between the 
other factors, was arbitrarily selected for use. 

This breakdown of the Action Planned category into 
three phases resulted in again expanding the number of bene- 
fit codes. Figure I9 shows the complete breakdown of the 
benefit codes into which a request could be categorized. 

3. Third Categorization, Quality of Dollar Estimation 

a. Estimable Requests 
Requests with dollar savings specifically identi- 

fied on the FESO Project Effectiveness Questionnaire (Figure 
l6, Question 5) were classified by Hendrickson and Fisher 
as estimable. If the identified savings were of the one-time 
type, versus recurring, the amount so identified was used as 
the request benefit. The benefit credited to CEL assistance 
for such a request was the request benefit reduced by the fac- 
tor for implementation probability and the information-%-factor 
as appropriate. If the estimable identified savings were of 
the recurring type, the request benefit used was the present 
value of the first five years of savings. A present value 
factor of 3.935 was used for a steady cash flow throughout the 
year, utilizing a l0 percent rate of return. A present value 
factor of 3.977 was used for this study to conform with cur- 
rent NAVFAC guidelines [Ref. l4]. Again the benefit credited 
to CEL assistance was the request benefit reduced by appropriate 


factors as done above. 
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BENEFIT 
ESTIMABLE 


Code 


Ol 
02 
03 
04 
05 


06 


BENEFIT NOT 
ESTIMABLE 


Code 


07 
08 
09 
३0 
T 
We 
3 
l4 


45 


Action 
Action 
Action 
Action 
Action 


Action 


Action 
Action 
Action 
Action 
Action 


Action 


BENEFIT CODE CATEGORIES 


Taken 

Planned, Funded 

Planned, Budgeted 

Planned, Planned 

Considered, T and E Required 


Considered, More Research Required 


Taken 

Planned, Funded 

Planned, Budgeted 

Planned, Planned 

Considered, T and E Required 


Considered, More Research Required 


General Information 


No Benefit 


puplicate Request 


Figure l9. Benefit Code Categories 
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b. Not-Estimable Requests 

Beneficial requests which did not have specifically 
identified dollar savings generally fell into areas where the 
benefit was in the form of improved operations, better morale, 
increased safety, etc. In the FY-72 and FY-73 surveys, bene- 
fits of requests of this type were left unquantified. The 
approach taken by Hendrickson and Fisher was that with the ex- 
peron of requests which fell into the General Information 
category, each request had an identifiable benefit even though 
it was not readily quantifiable in terms of direct dollar 
Savings. They felt that each could, in some way, be identified 
with a project, the magnitude of which was normally relatively 
easily quantified. 

At this point, Hendrickson and Fisher made an 
assumption upon which a major portion of their analysis of the 
benefit of not-estimable requests was based. They assumed 
that, in order to commit funds to a project, a decision maker 
must, whether he realized it or not, expect a return in future 
benefits which is some percentage greater in present value 
than the initial outlay. They allowed that this percentage 
Might vary from decision maker to decision maker, but further 
assumed that a reasonable value for this would be ten percent. 
This represented a minimum expectation. Many projects whose 
expected return exceeded this minimum would be undertaken, 
and, of course, the actual benefit of a project might fall 
short of the expectation and yield substantially less than 


ten percent. Nevertheless, a mean value of ten percent seemed 
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intuitively appropriate and consistent with rate of return 
guidelines for cost benefit analysis procedures. No fault 
could be found in this logic, and even though it appeared to 
be somewhat conservative, there was no logical basis for 
selecting another value. Accordingly it was used in this 


study. 
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APPENDIX E 


MODIFICATION OF INITIAL DATA GATHERING INSTRUMENT 


In l97l, procedures were formalized to monitor the flow 
of information between the field users and the laboratory as 
a means of developing information to measure the usage of the 
service by field units. A form was developed to gather inform- 
ation on assistance requests that were received and answered 
by telephone (Figure 20). Since the bulk of the requests are 
initially received by telephone, and many are answered by 
the same method, a method was needed to allow convenient docu- 
mentation and follow-up on their progress. To avoid placing 
an inordinate and unnecessary burden on research and secretarial 
personnel, that would have resulted if formal memorandum re- 
porting had required, the form was designed to allow comple- 
tion by the researcher in his own handwriting and was printed 
on carbonless paper so he could retain a record for his own 
purposes. This instrument provided a convenient method to 
record and track those requests which did not result in some 
form of written communications with the requestor, and satis- 
fied the information requirements of the researcher and the 
FESO alike. For example, the data on these forms was used 
as input to produce the performance measures previously dis- 
cussed and shown in Appendix A. However, efforts to evaluate 
Demetit would require obtaining certain additional informa- 
tion. It was considered that some of this information could 


best be gathered by CEL researchers during the initial call 
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RECORD OF RESPONSE TO VERBAL REQUEST FOR RDT&E ASSISTANCE 
NCEL (T) 2960/38 (7-7t) date 





INSTRUCTIONS FOR RECORD OF RESPONDING TO VERBAL REQUESTS FOR 
RDT&E ASSISTANCE FORM 


I. This form is for recording verbal requests for ROT&E Assistance that do not result in an Official letter being sent to the requestor. 
2. Fill in Dy hand using ballpoint pen, or type if you prefer, 


3. If request is satisfied during the initial conversation with the requestor, complete items I thru §. send original to LO3C and make appropriate 
internal distribution. 


4. If further action is required after initial contact, complete items } thru 5 and send copy to LO3C. Whoever is assigned action, complete item 6, 
send original to LO3C and make appropriate internal distribution. 


5. Comments or suggestions to imorove the form should be directed to LOAC. 


- 


l. Received by: 


8 
name code extension 


2. How received: (check one) telephone NCEL visit site visit 





3. From: 





name telephone 














fm i २? code activity 


4. Details of request: 











5. Was request satisfied during initial contact? Yes No 
a. If no, who will complete? 
b. lf no, estimated manhours to complete: 0-8______ 8-20 over 20 


name 








6. Nature cf service performed or information supplied (include dates): 























DATE 


e oe e 





Copy to: ; 
divisicn other 


Figure 20 Record of Response to Verbal Request for 
RDT&E Assistance 
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for assistance. Accordingly, the form was revised and is 
shown in Figure 2l. With the exception of format changes to 
increase the amount of space available for writing, only three 
questions were added. 

Question Number 5 was added to obtain information that 
would relate the request to a particular type of field pro- 
ject. This information would be used to calculate benefits, 
since construction or repair projects would result in one- 
time benefits, while maintenance and operating improvements 
would generate annual benefits that would have to be discounted 
to present values. 

Question Number 8 was included to obtain a better assess- 
ment of the approximate cost of responding to requests that 
were initiated by telephone. Accordingly the researcher was 
asked to indicate the approximate number of hours expended on 
a particular request. This information was not used in this 
study, but provides input to a data base for future cost/ 
benefit studies at a minimal collection cost. 

Question Number 9 requests the researcher to make a purely 
subjective estimate concerning the possible value of the 
assistance provided to the requestor. Since the form was not 
revised until December l976, this type of information was not 
available from CEL researchers for this study. However, it 
Will be available for future evaluations to guide the FESO in 
excluding specific requests from further evaluation on the 


basis that they do not contain measurable benefit. 
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RECORD OF RESPONSE TO VERBAL REQUEST FOR FIELD ASSISTANCE 


CEL (T) 3960/38 (72-76) Rev. Date Received 


l. Use form to record verbal requests that do not result in a letter reply to the requestor. 
2. Use ballpoint pen, or typewriter. 


3. If request is satisfied during the initial conversation, complete items | thru 9, send orig. to LO3C and make appropriate internal distribution. 


4. If further action is required after initial contact, complete items I thru 6 and send copy to LO3C. Whoever is assigned action, complete 
items 7 thru 9, send original to LO3C and make appropriate internal distribution. 


. - ՞ - = - ՞ क ՞ = ՞ ՞ ՞ = - - - - - - - - - - - - - - - . > = ՞ ՞ ՞ ՞ = - > - > 




















Leo Coa e eee 2. How Received 
Name Code Extention (check one) 
3. From: C Telephone 
Name Code Telephone 
O CEL visit 
՛ | i . . . 
Title Activity O Site visit 
त A ee 
A A € ip NENA NN aS 
5. Type of project: “ Construction Z Repair = Maintenance = Operation G General Info. 
icheck one) = Other 
6. Was request satisfied during initial contact? © Yes I No (answer questions below) 
a. Who will compolete?____ O b. Est. manhours to complete: ८ 0-8 Զ 8-20 
Name (check one) 5 Over 20 


7. Nature of service provided (include dates): 











ep i A a e 
— Am aawára—a ULU Ñ Ñ. Ñ Ñ Ñ BESTE NEE 





—  .. का 5 ¡5  .................. Q. LLLUUUUUUUULLILIIIKIIIIiIIIIIILIiIIIIIIiILIIIIIIIIII III e sp 
waw—uwmamama—Iaa I... —..J..LUUUUIIIILIIIIIILIIIIIIILi I IIiIiIiIiIiIiIi!iIIIIIIIIIIIIIIII i s —a sv s 
—wm..qaTwwma I w r ar rra s a TT A rr 9५% जा. 


3. Manhours charged ferrcteno.): 0 ॥। 2 3 4 5 6 7 8 9 0 ıı ı2 ı3 ı4 5 I6 I7 ı8 9 20 


. To help decide whether LOSC shouid obtain cost/benefit information from the requestor, please indicate your best 
estimate of the potential cost savings. 





= Lass than $500 = $500 - $5,000 = $5,000 - $25,000 © Over $25,000 
GOO A «Other: त U U uuu Date Completed 
Division 


FE Ure 2] Record of Response to Verbal Request for 
Field Assistance 





APPENDIX F 


TYPICAL SAMPLING CALCULATIONS 


Mean: The sample group mean of the value X is given by 


— L Y 
X = —- X. 
8 > Dr 
i=] 
where X, = benefit of individual request 


n number of requests in sample group 


Standard Deviation: The sample standard deviation is 





glven by 
Հ 
S = 
where ո - l = degrees of freedom 
Confidence Limits: When small random samples (n 30). of 


Size n are taken from a normally distributed population, the 
sj CT pas the Student's distribution with nm - -. 
degrees of freedom. The 95% confidence limits for "t" are 


given by 


արշ. ..- 


Հ 
Տ 
2979 
n 


where ee is the confidence coefficient for 
which 2.5% of the area lies in each 
tailors he ԱԻ “GiseriToulelLon. 
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APPENDIX F (Continued) 











Multiple Group Sampling ~ Group No. l: 4 requests 
x 
i 
2350 
0 
8,380 
0 
2 = 9,530 
ar 97530 
X = 382.5 
4 


Sampling Trial No. l: 3 groups of 4 requests each 


Group No. X; X ca 
ji DGS 5,676,306 
7,500 
0 000 
23,862 
30,000 
2 PI 550 s 466,249, 690 
9,943 
0 
0 
39,770 
3 57455 54,455,284 
more Cole ONE 
DE 20 
2, 34 
3 
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APPENDIX F (Continued) 


Sampling Trial No. Լ (Continued) 











A SMD MOS © = 
S = 3 6 
2 
SN 
5 
E = — 
95 eos 
n- il 
9,607 
ee 
3 
ET: 


Limits of 95% Confidence Interval (Table III) 


Mean Benefit: 


Lower Limit = l2,l3A = 23,868 = —ll,734d 

peer Limit = | 2 | 3407 ८ ) 300७ कै 36,002 
Total Benefit: 

Lower Limit = -ll,734 x ll5 = -l,349,4]l0 

Upper Limit = ३36,002 x ll5 = ,240, 230 


ROI for investment of $53,766 


“,349,420 
Lower Limit = ———— = -25 
53,766 
a Lave 23 6 
Uoper Limit = ———— = // 


23,208 
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APPENDIX F (Continued) 


Grouped Sequential Sampling - Trial No. 8: म२8 requests 


Group No. X; եա)» 
ES O 2 2 2 30 
0 0 
8,380 70,224,400 
0 0 
2 37 500 306;250;000 
l5,000 225,000,000 


237992 569,395,044 
30,000 900,000,000 


3 9,943 98,863, 249 
0 0 

0 0 

39,770 “26652, 900 

4 0 0 
8,3802 0 

0 0 

2329 Sa, sc! 

5 0 0 
0 0 

0 0 

oO 0 


Db — 945,844 27752 o4, 964 


*repeated request 








= १45,844 
X= ——— = 8,202 
१8 
(armen Sea 
(445, 844) 
II O ees 
S= ı8 
जा A EE D 
2, 
go ee) 
tgs न 8,202 -- =o NG INO T ple EY 
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23 


24 


25 


26 
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28 


29 


30 
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Report 


ReBort 


Report. 


APPENDIX G 


ASSISTANCE PROGRAM BENEFIT 


REPORT SPECIFICATIONS 
LIST OF FIGURES 

Specification Original Estimate of Total 
Benefit - Report $+l2 

Specification Breakdown of Evaluation 
Status ~- Report l3 

Specification Breakdown of Field Evaluation 
Data - Report #34 

Specification Comparison of Contribution and 
Information Provided - Report #35 

Specification Summary of Savings by Type of 
Savings - Report +l6 

Specification Estimated Minimum Benefit of 
Evaluated Requests ~- Report l7 

Specification Summary of Evaluated Benefits 
by Type of Savings - Report #28 

Specification Statistics for Evaluated Bene- 
fits - Report +l3A 

Specification Summary Of Benefits by Benefit 
Codes - Report l9 

Specification Estimated Minimum Remaining 
Benefit by Benefit Estimate - 
Report #20 

Specification Comparison of Benefit Estimate 
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ASSISTANCE PROGRAM BENEFIT 
REPORT “SPECIFICATIONS 


REPORT TITLE: Original Estimate of Total Benefit 
REPORT NUMBER: 322 
PURPOSE: This report provides an original estimate of total 
benefit from all requests received from shore activities 
during the reporting period. It is based on the estimated 
ranges of potential benefit of each request without prior 
knowledge of field evaluations. It will be used to roughly 
estimate total benefit before field evaluations are started. 
INPUT PARAMETERS FROM EACH REQUEST: Estimates of Benefit. 
SELECTION CONSTRAINTS: 
l. Include all requests from shore activities. 
2. Include all requests for selected quarter. 
DATA MANIPULATION REQUIREMENTS: 
l. Sum and print the number of requests in each range: 
a. Zero dollars 
De From l to 499 dollars 
ca From 500 to 4,999 dollars 
dE com 5,000 to 24,999 dollars 
e. Over 25,000 dollars 
2. Calculate and print estimated benefit in each range. 
Multiply number of requests in each range by mid-value of 
respective range. Use 25,000 for "over 25,000 dollar" range. 


3. Sum the estimated benefit in each range.’ 


Figure 22. Report Specifications - Original Estimate of 
Total Benefit 
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OUTPUT PARAMETERS: 
l. Number of requests in each range 
2. Estimated benefit in each range 
3. Total estimated benefit 


FREQUENCY: Quarterly 


Figure 22 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 

REPORT TITLE: Breakdown of Evaluation Status 
REPORT NUMBER: ]3 
PURPOSE: This report summarizes the number of requests received 
from shore activities by whether and by whom an evaluation has 
been performed, by the estimated range of potential benefit. 
It is used to show the status of evaluation efforts for the 
reporting period. 


INPUT PARAMETERS FROM EACH REQUEST: Evaluation, Estimate of 
Benefit 


SBEEECTION CONSTRAINTS: 

l. Include all requests from shore activities. 

2. Include all requests for selected quarter. 
DATA MANIPULATION REQUIREMENTS : 

l. Identify and sum the number of requests in each evalu- 
ation phase. 

2. Identify and sum the number of requests in each range 
of estimated potential value. 

3. Provide a matrix of Estimate of Benefit and Evaluation. 


OUTPUT PARAMETERS: Number of requests in each of the above 
categories. 


BeeOuUeNCY: Quarterly 


Figure 23. Report Specifications - Breakdown of Evaluation 
Status - Report #l3 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 

REPORT TITLE: Breakdown of Field Evaluation Data 
REPORT NUMBER: 24 
PURPOSE: To determine the number and type of responses to the 
feedback questionnaire used to evaluate requests from shore 
activities. This report provides background information for 
directing future program efforts and effectiveness measurements. 
INPUT PARAMETERS FROM EACH REQUEST: Estimate of Benefit, Any 
Benefit, No Benefit, Benefit Code, Use of Advice or Assistance, 
Type of Saving, Contribution of Advice or Assistance to Decision, 
Percent of Information Provided by CEL Compared to Total, Would 
Solution be Applicable to Other Activities. 
SEE TION CONSTRAINTS: 


l. Include all shore activity requests 


2. Include all requests for selected quarter 
3. Include all requests which have been evaluated in the 
field. 


DATA MANIPULATION REQUIREMENTS: 
i. @denciry and sum responses for the above parameters. 
2. Determine frequencies for identified responses. 


OUTPUT PARAMETERS: Number of requests in each of above 
categories. 


FREQUENCY: Quarterly 


Figure 24. Report Specification - Breakdown of Field 
Evaluation Data - Report #34 





ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 


REPORT TITLE: Comparison of Contribution and Information 
Provided 
REPORT NUMBER: ]5 
PURPOSE: This report summarizes the field evaluation data 
provided in response to questions on requests received from 
shore activities related to the contribution of CEL assistance 
and the percentage it represented compared to the total 
required to make the decision. It will be used to determine 
the correlation between the two parameters in a later attempt 
to establish reliable factors for each of the Contribution 
categories; 
INPUT PARAMETERS FROM EACH REQUEST: Contribution of Advice or 
Assistance to Decision, Percentage of Information Provided by 
CEL Compared to Total. 


SELECTION CONSTRAINTS : 


l. Include all requests from shore activities. 

2. Include all requests for selected quarter. 

3. Include all requests which have been evaluated in the 
field. 


DATA MANIPULATION REQUIREMENTS: 


१. Identify and sum the number of requests in each 
Contribution category. 


2. Identify and sum the number of requests in each 
Percentage category. 


3, Provide a matrix of Contribution and Information 
Provided. 


Figure 25. Report Specifications - Comparison of Contribution 
and Information Provided - Report l5 
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OUTPUT PARAMETERS: Number of requests in each of the above 
categories. 


FREQUENCY: Quarterly. 


Figure 23: 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 

REPORT TITLE: Summary of Savings by Type of Savings 
REPORT NUMBER: 326 
PURPOSE: This report provides the savings for requests from 
shore activities during the reporting period which have been 
evaluated in the field, by the type of saving. This informa- 
tion will be used to indicate how savings from assistance pro- 
requests are distributed among the various types of savings. 
INPUT PARAMETERS FROM EACH REQUEST: Type of Savings, One- 
Time Construction or Repair Cost Saving, Annual Maintenance 
Cost Saving, Annual Operating Cost Saving, Project Cost. 
SELECTION CONSTRAINTS: 

l. Include all shore activity requests. 

2. Include all requests for selected quarter. 


3. Include all requests which have been evaluated in the 
field. 


DATA MANIPULATION REQUIREMENTS: 
l. Identify and sum the savings in each type of savings. 
2. Identify and sum project costs. 
3. Calculate the mean saving for each type of saving. 
4. Calculate the mean project cost. 
OUTPUT PARAMETERS: 
l. Number of requests in each category. 
2. Total savings for each type of savings. 
MOTELS! project cost. 
4. Mean savings for each type of savings. 
5. Mean project cost. 
FREQUENCY: Quarterly 
Figure 26. Report Specifications - Summary of Savings by 


Type of Savings ~ Report slo 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 


REPORT TITLE: Estimated Minimum Benefit of Evaluated Requests 
REPORT NUMBER: ]7? 

PURPOSE: This report provides an estimate of minimum total 
benefit for requests from shore activities during the reporting 
period, which have been evaluated in the field. It is based 

on the estimated range of potential benefit for each evaluated 
request. It will be used to determine the accuracy of estim- 
ating techniques and to determine the mid-level value for 
requests with benefits over $25,000. 

INPUT PARAMETERS FROM EACH REQUEST: Estimate of Benefit. 


SELECTION CONSTRAINTS: 


l. Include all requests from shore activities. 
2. Include all requests for selected quarter. 
3. Include all requests which have been evaluated in field. 


DATA MANIPULATION REQUIREMENTS: 


l. Sum the number of requests in each range of estimated 
potential benefit. 


s | lace and print estimated benefit in each range. 
Multiply number of requests in each range by mid-value 
of respective range. Use 25,000 dollars for “over 
257000 dollar” range. 


3. Sum the estimated benefit in each range. 


Figure 27. Report Specifications - Estimated Minimum Benefit 
of Evaluated Requests - Report l7 
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OUTPUT PARAMETERS: Number of requests in each range. 
l. Number of requests in each range. 
2. Estimated benefit in each range. 
3. Total estimated benefit. 


FREQUENCY: Quarterly. 


Figure 27 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 


REPORT TITLE: Summary of Evaluated Benefit by Type of Savings 
REPORT NUMBER: 28 
PURPOSE: This report provides the benefit of requests from 
shore activities during the reporting period, which have been 
evaluated in the field, by the type of saving. It is based on 
the savings or project cost provided by the user. It will be 
used to indicate how assistance benefit is distributed among 
the various types of Savings. 
INPUT PARAMETERS FROM EACH REQUEST: Type of Saving. 
SELECTION CONSTRAINTS: 

Y. Include all shore activity requests. 

2. Include all requests for selected quarter. 


3. Include all requests which have been evaluated in the 
field. 


DATA MANIPULATION REQUIREMENTS: 


l. Calculate and print benefit for each benefit code and 
corresponding type of saving. 


2. Sum the benefit for each type of saving. 


3. Calculate mean benefit for each type of saving and 
cotal benefit. 


OUTPUT PARAMETERS: 
l. Number of requests for each type of saving. 
2, Benefit for each type of saving. 


3. Mean benefit for each type of saving. 


Figure 28. Report Specification - Summary of Evaluated 
Benefits by Type of Savings - Report #+48 
Page 7 ०£ 2 
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4. Total Benefit. 
5. Mean of total benefit. 


FREQUENCY: Quarterly 


Figure 28 
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llo 





ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 


REPORT TITLE: Statistics for Evaluated Benefits 

REPORT NUMBER: ]8A 

PURPOSE: This report provides statistical analyses of the 
benefits of requests from shore activities during the reporting 
period, which have been evaluated in the field, by the type of 
Saving. It will be used to determine trends, if any, which 
exist between various reporting periods in the distribution 

of benefit among the various types of savings. 

INPUT PARAMETERS FROM EACH REQUEST: Type of Saving 

SELECTION CONSTRAINTS: 


l. Include all shore activity requests. 


2. Include all requests for selected quarter. 
3. Include all requests which have been evaluated in the 
field. 


DATA MANIPULATION REQUIREMENT: The SPSS procedures for range 
tests, which include the student "t" distribution, are used. 
Although these tests are not ideally suited for testing the 
significance between the sampled groups of benefits used en 
this study, they will at least provide an initial indication 
of the existence of any trends, when compared to the results 
of future evaluations. 
OUTPUT PARAMETERS: 

l. Analysis of variance. 


Figure 29. Report Specification - Statistics for Evaluated 
Benefits - Report #l8A 
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2. Significance of ranges and the differences between 
the mean benefit for each type of saving. 


OUTPUT: Quarterly 


Figure 29 
Page 2 of 2 
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ASSISTANCE PROGRAM BENEFIT 
RERORTIFSPECIFICATTONS 


REPORT TITLE: Summary of Benefits by Benefit Code 

REPORT NUMBER: 29 

PURPOSE: This report provides the benefit of requests from 
shore activities during the reporting period, which have been 
evaluated in the field, by the type of benefit. It is based 
on the savings or project cost provided by the user. It will 
be used to indicate how benefit is distributed among the vari- 
ous types of benefit. 

INPUT PARAMETERS FROM EACH REQUEST: Benefit Code. 

SELECTION CONSTRAINTS: 


l. Include all shore activity requests. 


2. Include all requests for selected quarter. 
3. Include all requests which have been evaluated in the 
rrelds 


DATA MANIPULATION REQUIREMENTS: 


l. Calculate and print benefit for each benefit code and 
corresponding type of savings. 


2. Sum the benefit for each type of benefit. 


3. Calculate mean benefit for each type of benefit and 
total benefit. 


OUTPUT PARAMETERS: 
l. Number of requests for each type of benefit. 
2. Benefit for each type of benefit. 
3. Mean benefit for each type of benefit. 


Figure 30. Report Specifications - Summary of Benefits by 
Benefit Code - Report +l9 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 


REPORT TITLE: Estimated Minimum Remaining Benefit by 
Benefit Estimate 


REPORT NUMBER: 20 
PURPOSE: This report provides an estimate of the minimum 
benefit of requests from shore facilities during the reporting 
period which have not been evaluated in the office or the field. 
It is based on the estimated range of potential benefit for 
each un-evaluated request. It will be used to estimate re- 
maining benefit, which when added to evaluated benefit will 
produce a more accurate value of total benefit after evalua- 
tions have started. 
INPUT PARAMETERS FROM EACH REQUEST: Estimate of Benefit. 
HE EE CT TON CONSTRAINTS: 

l. Include all shore activity requests. 


2. Include all requests which have not been evaluated in 
əffice or field. 


3. Include all requests for selected quarter. 
DATA MANIPULATION REQUIREMENTS: 


l. Sum the number of requests in each range of estimated 
benefit. 


2. Calculate and print estimated benefit in each range. 
Multiply number of requests in each range by mid- 
value of respective range. Use 25,000 for “over 
22000, collar” range. 


3. Sum the estimated benefit in each range. 


Figure 3270. Report Specification ~ Estimated Minimum Remaining 
Benefit by Benefit Estimate - Report #20 
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OUTPUT PARAMETERS: 
l. Number of requests in each range. 
2. Estimated benefit in each range. 


3. Total estimated benefit. 


FREQUENCY: Quarterly 


Figure 3l 
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ASSISTANCE PROGRAM BENEFIT 
REPORT SPECIFICATIONS 


REPORT TITLE: Comparison of Benefit Estimate and Benefit 
REPORT NUMBER: 23 

PURPOSE: This report summarizes the benefit of requests 
received from shore activities during the reporting period 
and estimated benefit of those requests. It will be used to 
determine the correlation between actual and estimated bene- 
fit within ranges of benefit. 

INPUT PARAMETERS FROM EACH REQUEST: Estimate of Benefit. 
SEILEBCTION CONSTRAINTS: 


l. Include all requests from shore activities. 


2. Include all requests for selected quarter. 
3. Include all requests which have been evaluated in the 
field. 


DATA MANIPULATION REQUIREMENTS: 


l. Calculate and print benefit for each benefit code and 
corresponding type of savings. 


2. Recode benefits into range of estimated benefit. 


3. Identify and sum the number of benefits in each range 
of estimated benefit. 


4. Identify and sum the number of requests in each range 
of estimated benefit. 


5. Provide a matrix of Benefit and Estimate of Benefit. 


OUTPUT PARAMETERS: Number of requests in each of the ranges 
of estimated benefit. 


FREQUENCY: Quarterly 


ire 32. Report Specification - Comparison of Benefit 
Estimate and Benefit - Report 2l 
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APPENDIX J 
ASSISTANCE PROGRAM BENEFIT 
OUTPUT REPORTS 
Assistance Program Benefit, Original Estimate of Total 


Benefits - Report +l2 


Assistance Program Benefit, Breakdown of Evaluation 
Status - Report $l3 


Assistance Program Benefit, Breakdown of Field Evaluation 
Data - Report १24 


Assistance Program Benefit, Comparison of Contribution 
and Information Provided - Report *+l5 


Assistance Program Benefit, Summary of Savings by Type 
of Savings - Report +l6 


Assistance Program Benefit, Estimated Minimum Benefit 
of Evaluated Requests - Report $+l7 


Assistance Program Benefit, Summary of Evaluated 
Benefits by Type of Benefit - Report *l8 


Assistance Program Benefit, Statistics for Evaluated 
Benefits - Report ४282. 


Assistance Program Benefit, Summary of Benefits by 
Benefit Codes - :२२€८०:८ १०29 


Assistance Program Benefit, Estimated Minimum Remaining 
Benefit by Benefit Estimates - Report #20 


Assistance Program Benefit, Comparison of Benefit 
Estimate and Benefit - Report #2l 


256 





T 
eu 


(६ 
(५ 
(at 
(54 
(6! 
(०२ 
(SnI 


# Jaodey - szTyoueg Te}OL JO 9IPUTAISA TRUTBTAO ‘ATJeuo_ wethoitig SOUPBISTSSY pp Sandra 


u ad 


0 
0 
n 
0 
0 
0 
90१396$१६१”" 


JINVIANYA 


0 
0000°00NS2 
oonn* 00NST 
N000°NSL2 
n000°0S2 
n 

R60°EF NO GOLF Unth 


333202000570 


A30 als Ny 34 


0 
000००००१५९१ 
0000” 0000L2 
0000* 06002 
0000"0ՕԼհ 

0 
0000°005009 


ԽՈՏ 


ee ՏՎՈԼ ԱՎ ԿՂՊՈՎՈԺՑՔՈՏ 40 


ԻԼ e S38v49 ArviOl 

०० ISILANG 

०० IS3iL4Nqa 

մի IS3i 4NA 

"$ ISFL4NQ 

ne {SJL ԵՔ 

= ISJL4NA 

NNTIVANINA INTINI 809 

ԴՀԳԾՂ INTA 360) TIAVIMVA 
tr49nqaQ 4D AlLyWTiS3 ISJ 4aNẹ AN NMOO NINONA 

4749N439 । १५४ JIAVINVA NNINILTYD 


N f I լ a t H 9 S 3 0 տ e a a a क 7 का का का ռա 0 տ տ ա 





T# qaodaey - snjeqjg uorqaenreAsq JO uMOPpyeotg ‘ATyousq werhorg DOUBISTSSV 
0°00१ L°02 o° €°@h WwiOl 
ort 0६ Sh OL NWNIOI 

Joononnsnajananaanajacnacnaosa]o 
T! v'e I 0 I O I 
I 7०१) Լ 0 I 0 I 
n’f Լ 0"00 Լ 0 I 0 I 
6 Is I 0 I 0 I ° 
ք«»»»օթօօալաօօօօօօօլաօաօօաթաաավօ 
I 0 Late անին լ 
I n ... I 6°? I 
n° ç I 0 I 0°09 ] 020१ | 
५ I 0 If I 2 I S$ 
Լ»»»»օ«օօթալ»տ»օօթօօօաաօավտաաօ»թաա»ռավ»օ 
Io I S°S ԱԹ I 
I 0 t e'it | (7772 । 
h°?t Լ 0 Լ եհ լ 09:65 | 
eT I 0 I 8 I of Inh 
[००००७ ७७ हूँ ० ७७ ७ ७ ७ हूँ ७ ७ ७ ७ ७ ७७ ० [ ७ 
Լ 0 I 9°9t |] ४०११ I 
I 0 I €°€S ] 00 I 
"ՈՏ T Ո I ७°2६ I 8°49 I 
£ ८ I 0 I h2 I 6h I £ 
[००००७ ७ ७ ७ हूँ ७ ७ ७ ७ ७ ७७ ६ ७ हूँ ७ ७ ७ एक ७ ७ ७ ७ ५ ७ 
T o I 6°9 RE I 
I 0 T ९४९९ eet ] 
I 70 I 9°2S I n°¿ih I 
6! I 0 I of I 6 I °? 
[००००००००[००००००००[००००००००] » 
Pe a I O I 0 I 
I "६8 TFT 0 I 0 լ 
Փ"ՎԼ Լ °00t [| 0 I 0 լ 
५२ I ०२०२ I 0 I 0 7) 
[== s. .... ........[........Ñ........ ABILING 
॥ ८ । 200 I 0 I t3d 404 
TWwint I 43d I0I 
ԿՈՍ 66h OL | 0 XNYTRI LId MOY 
I ınnn) 
१११३ 


տ फक ह क 9 9 9 कक हैं क ह ह Ժ 9 क हैं छ 


NOTLYN TVA AVAJ AR 
ԿՓ Փ क 30 nolIıvına vi 


श 9 9 क NN, ४ झ $॥ մ յ 9 9 9 9 ցս 9 Ա 9 


ATJINTA 40 JLVWIIE3 LS3i4N9 


w २ है 9 9 9 हक 9 ց ԱԽ 9 9 9 ७ 9 ց ս झा 


Bs FOU) 


Y" 


ՇԵ 


२4007 न्‌ 


5 





6 39 [_ abea 


PIt yzaodsy - 3T39U98 JO 9JPUTISH 


‘pjeq UOTAIENTRAY PTeta JO umopyeorg ‘sqtTyouseq wearaborg ə9Əocueqstss op sıanprı 
n SICYI ANTSSTw Օհ 838v9 OT IVA 
OI nennt Sn Wind 
O° ont ¿eo (“० ç S 
OV 
९०९७ Q°/ Ք"(| Q հ छ 
१°५८ ç °$ G FES ne $ 
टट २०२२ 2° 22 Ot 2 
( | ५३293 ) (IN3IN3IA) CUNADHAd) AINIAGMANA = FAN) VAEVT AMNNALID 
MIMI fay ԽԴՉԽՀՈՊՉ«Վ ANNINPIAI FJLNIASAY 


ՉՈԼԼԺՂՈԿՂԴ AILSNFOYV JATIN 


AT4AANAA 30 Jtywtls4 ISIL INA 





‘eqeq uoTReNTeAY 


0° oot 
6°56 
CIN@AIMAd) 


9343 [IV 
JATIVNWOND 


6 JO 


Z obed 


ԵԼ qaodey ~ Ftyoueg Auvy 


PISTA Jo umopyesiag *s3tjouog werhorg DOURBISTSSY 9p DdANDTA 


0° ont 


L° Q 
१०१6 
CINJDMI9Ad) 


AJNINM AM 4 
ՈՀԼՏՈՐՈՄ 


onol 


१०१७ 
(INJINAd) 


AININMANHH 
SATIVV9H 


५४३९०) 3INTSSIW Gh 2384) ՈԼՀՄՈ 
Sp Wink 
ç Հ ON 
en լ 834 


AININMINY 3009 WANT ANUS 


JIN VWSAY 


LIAIINIAN ANY LINAANY 


न ७) 








¿oeaeq uoraenTeAq पारा JO प्र७0ए0अणएउयप्त “इयावउउफप््घ एएयरिण्यत SDUeISTSSY 


O° ont 


O° ant 


(IN3)H3d) 
१3५4 OV 
IATIYTINWAD 


O° OT 
ONTSST a 
$° çç 
$ ° $ $ 
ç ° ç$ 
(IN32)343d4) 


ԽԴՎՅՈՊՀԿՎ 
qg3aisnfarw 


6 30 


en 


O° ant 
$६७ 
CIC 
Cre 
ee 

( լ"ՀՉ"ՀԴ) 


AINANM AN 4 
3AJ IN TH 


ՏՎՏԾԴ 


Vit 3-200कप्त - rzəuəg ON 


Gn 


en 


AJMANM IN 4 
ԶԱՈՂՐՏՈԴՄ 


yn Tectu 


Win) 


3002 


9p sanprı 


१ 09०४ ՈԼՂՎՄՈ 


ՁՋԽՄԿ ՎՈ IND 
33807 znata 
924734 I0N Ü d3NI 
AVAV TI VAY CANT ON 


VAAVT ANNONAILVI 


IT 4ın JR ON I ANHON 


rel 





6 JO + obeg 


/# 32०१३ - əpoo ATyouag 


*peaeq սօլգծոլեռպ յթլծւզ յօ ս'ճօքվաօմզ '54լգյծսօզ աօ60զ Յօսծյտւտտ 9p SANBTA 
0 ५3५) ONISSTw sn SISYI ATIVA 
0° ant 0° 00 Gh Wwini 

n° 50 er CI 5 ति bl LI44GngR ON 
१०१6 ron n° Ob 8| Fl NINT WANING 
E" ES 2°? ՍԱՄ լ el Վ ?OINJIAOISNOI NOLLIY 
[us L° q fo £ 8 CONN 4 §QQNNY Td ԽՈԼԼ27 
n° nh 0°02 n* 02 b í S4 ION n3y4vi NOTI9¥ 
h° te ट्र CC | ० է "'J3833ISN0D02 NOTLIY 
e.o (०० ("9 ç 2 ANNA “OINNVTA NDTLIN 
१००३ ००५३ ००५३ / լ VWILSA *NIMYV] NNIJIY 
(N3 JH3d) CINADHAd) (ԼԵԴ) AINANMIAMY 300) १39४१ AMHNZLVI 

03844 ray AININHANA AININNAMA JINTISAY 


3ATLvINwNN ՈՅԼՃՇՈՐՈՄ 34lıvaın 


23009 LTIYIINAR ANILINA 


OZ 





6 Jo g obed 


ԵԼԱ Հ:ԾՎՅյ - soOUeASTSSY AO 99TApy JO ƏSN 
’'e3jeq սօլգծոլաճգ PTSTI JO umopyesrzg *SITI9U9SE weiAboiIgd SDUeJsTssy oP oanbty 


0 Տ3ՏՄԶ INTSSIW Ոհ S3Sv9 ՈԼՂՄՈ 
Van} n° Any Gn Twink 

a° ont eeh em Ë Pr : 39Nv3 30 TAN 
O° On sul ०००३ ८ IS ԿՀԵԼԸ 
5°28 SI 9° Gl L Ir NUTIYWNOJNI ԴՅԱՅԿԽ39 
१००० Smg ՀԵՇ | २१ ISNNI NMIIVINIMITAWIT 
5°2१ ն ("օգ $ I$ ISNNI NOTLYINIWITAW] 
N° SS ५०२३ Tout S ie CILVINAWSTdWt LIVANIA 
sten S°?h ԱԴ) { I OAINAWATVAWE ATLIIYIO 
(६(५००५३४) CINJINAd)  CIN3I8II)  AINANDI=4 3009 १3६४५7१ 4५09१३।४9) 

।) 39११ 0४ ADNANMINY ADNANMANY 3LNNNSHY 


SATIVINWND ՈՀԼՏՈՐՈՄ SJATAV VIN 


JSINVAISTSSY AN 3)TAOY YN ՀՏՈ JSN 


ios 





Dm 
U= 
O 
N O 
U 
OV 
© 
ar 


PC է ԴՀօՎԾԱ - sbhutaes jo odAy, 


mY 


“e3eq UOTIPNTPLAH PISTA JO UMOPAR9A ‘sqytTyJousq werborAg DOUPAISTSS ob sınbrı 
$I S3Sv2 ONISSTHW २६ Տ357Դ GIVvA 
0°00} O°OOT Sh ՂՄԼԸԼ 

0९00] ANISSIN Ce छा 39४४४ 40 LNO 
o°not T.a? १०02 ७ ० Y3HLO 
९०१८ S° / $ । "५२ at b 37evwWIlS3 LON 
n° ne. ռ"6 ("Գ ç ç 0 TWANNVY NI  SINIAVS 
0००२ नि"७३ £६१ 9 e VW ԴԺՈԽԿՄ NI ՏՑԶԱԼՈՄՔ 
¢°9 ç ° 9 b° n टे լ ՀՈԼ 3NO NI SINIAVS 
CIN39H39d) (IN3)N3A) (INA)N3IA) AININNINI 3009 TWAVT ANDONNSLYVI 

१3४3 FAY AININOINIS A2N3nn3HdH3 23INAVNSAV 


FATLYINWNDI OJLSNFOY ELUSE BER, 


INTAYS 40 3dAl AVSdAl 


३64 





0007 

O°00T 

9° Rg 

g° ns 

१०३२ 
CINIINGd) 


9344 CAY 
SATALVINWNI 


6 JO L abea 


VI# 3adodəu - SDUeISTSSY AO DOTAPW JO UOTINATAFUOY 
’e3jeq uorgenTeAN PTSTA JO umopyeorag ‘sqRTyouog werbhorg 9DUPISTSS 7 olnbt, 
լ Լ Lv! A լ ! Մ 0०४ [तु 


ol S3SV9 £NTESIW ՕՏ 53690) ATIVA 

N° 004 n° OCT Sh Iwind 
ON 222 0. ३9४४५ 40 LNO 
rial ७०9 հ հ LVONSWHNIIN Q3gIAOHd 
308 ("०२ el ç 4 03€N MANT 03F0TAONA 
Les? 0° 02 b € A WMA LI3I35 AIII3H 
9°82 ८0 OT ն H3IlHYV3 nv OIWHTINOI 


CAINIINIAd) (])N3OM3d) AINANNINA 3003 
AININMAN4 AINANMANA ՊՀԼՈՂԸՑՅՄ 
03ısnrav JATIVIIN 


VWAVT ANOOSILVI 


AL IINVISTSSY Y! JITAUVY 30 NOTLART=MINOI RAIHLNOI 


Les 








6 JO p abea 


#T# 3aodsy - THD Ag pPpepTAOAg uoTAeWAOFJULT JO əbequəodzəd 
“e32q UOTIBNTRLAH PISTA JO umopyeorag ‘sqTjyoueg werborig soueISTssw gp sandra 


n S3५४) S9NISSIWM Le 535") QIYA 
A°adt n° 00t Շի Wildl 
oroar NIS ՅՈ ३9४४४ 30 LAN 
0°00f Po b° n ծ OOT 
9°26 Las २०२ | 06 
ծ" 28 ant 6°R հ 08 
T*hl ie p 2 04 
८०११ ST IE, S 09 
tq Rent 6" 8 հ 09 
१०९१ ("१ हट I on 
9°62 S'AI Tai ० 05 
ai । (०० $ 02 
(IN394 3d) (INFIN34) (IN392H34) AININOHINY 3003 VISVT ANXNONILVI 
NINA PAY AINANMAMN 4 AININMAMA ALIN WNSAY 


FATIVIAWNI ՈՀԼՏՈՐՕԽ JATIV VG 


) "39 ARM Ո`ՎԱԼՈՈՎՎ NOTIVWHOINT 40 INIDNSd OINI 


T66 








pI ३-०099पत -— SOTITATIOY AOUYI0 OL 
'ezeq uotIenTeAayg PISTg JO UMOPABDIH 


n’09nT 

aront 

հ" Լ6 
(INFIHIA) 


NINA FOV 
JATIVAINWNSI 


O° Ont 


ONTSS Tu 
9°? 
հ" (6 
(IN3)A34 


AININAIN 
ԴՀԼՏՓՏՈՐՕՈՄ 


) 
4 


ԴՄ 


6 JO 6 ebeg 


५ ՇՀՎՏԾԴ AnTSSIn 6¢ 
O° OCT gn Wink 
१”१ | १ 
2° 2 | टे 
rra HS լ 
(|ԻՉԴՎՀՎՁ) AJNANNJN4 340037 
AINANMANY ՊՅՀԼՈՂՈՏԵՄ 
JATIV794 


“3dın nt I3Iav9Tidav 39 NOTINTOS OANOM 


etqeottddy əg UOTINTOS PTNOM 
s} trzjəuəg werborg SIUeISTSS 


99 sınprı 


sasr) OTIYA 


ՅՋԿՄՎ ՀՈ LAN 
ON 
SIA 


११४७१] AAONAAVO 


3ISNAHILO 


400 ॥ 





I 


0°00 
le 


Ց0 | 


GT# ԴՀՕՎՅԱՎ - քռքւլճոօգ սօւգօաօյսալ 
pue uorInqTA}uo) Jo uostaeduoy ’ITJ9usg weaboıIgd DOUPISTSSY 


Lb sandra 


AT a SNOILYAXJSAO INTSSIW JO N3INNNN 
ն h° ` L° € 9०१! ned S'el "հել ¿ € sol vH Viol 
2 յ հ 2 S հ լ ० ç NWNIO0I 
Jecsananaajanaananafanaanaaiacananaj annnaanaj]anannanaj]nnncanaaj annnananjannanaaa]a 
Np Lt I "१ I ("4 I 0 I 0 I "१ I 0 I 0 I 0 I 
Լ 0"05 I 0"00 [I 0९७2 I 0 I 0 I 0००९ I 0 I 0 I 0 I 
I 0°S2 ] 0°62 I 0íG2 I 0 I 0 I 0°१2 I 0 I 0 I 0 I 3WWOISY OI0TAONA 
t i t | I ! I 0 I 0 ա I 0 I 0 I 0 In 
[००००००००७ ७०७७७ ७ ७ ७ # हूँ ७७७७७ ७ ७ ७ ७ हूँ ०७७०७ ७ ७ ७७ ५ ७७७ ७०७७ ७७ ९ ७०० ७७७ ७ ७ ७७ हूँ ७०७७ ७ ७७ ७०१८७ ७ हूँ ७ ७७ ७७ ७७८७ ७ हूँ ७०७ ७ ७ ७७ ७ ७ || ७ 
I O I 0 I 0 I ned Kusel հեչ Il 0 ՒՆ O II L's I 
I 0 I 0 I 0 I o0°O00tT ? 0'09 ££ 0९06 I 0 r 0°09 JY £°€€ fT 
I 0 ? O I 0 Լ ՀՅ Լ 442  2@°et I Q I 2:46 I 3°%6 I sn NANT Q30IAONd 
I 0 I 0 I O I # I ç ) € I 0 Il 4 լ | I | 
[०००००००७ ७ ५ ७०७७७ ७७ ७ [ै ७७७ ७७ ७ ७ ७ ५ूँ ७ ७७७ ८७ ७७ ७ हुँ ७ ७७७७ ७ ७ ७ हूँ ७ ७ ७ ७ ० हूँ ७००७ ७ ७ ८ ७ हूँ ७ ०८७ ७ ७ ७० ८ 'ुँ ७०७ ७ ७ ७ ० ७ ७ मु ७ 
| Ee ae | I O I ett I 0 I 0 ա... լ ('4 I ("4 I 0 I 
I 0°0S I 0 I 0°S/ I 0 I O I १००९ I o°oot r 0°02 I 0 I 
I €°ht J 0 I ७००१ I 0 I 0 ID Cpt TOT PI Pe ett. TO I 48 493798 034973H 
I ft I O | I 0 I 0 ան լ | ա I 0 I Հ 
I==......]........]........]........-........].-.......]........]........ ]........[. 
I 0 I 0 I 0 I 0 I nee I 0 I 0 I L'S I neh I 
I 0 I 0 I 0 I 0 I 0°0r I 0 I 0 I 0°02 I tO [I 
I 0 I 0 I 0 I 0 I 0०0१ I 0 t O I 0°02 I 0०0१ ? UY3 NY OINNIINOS 
I 0 I 0 I 0 I O I 2 I 0 t 0 I । | ? I Í 
Less. ....]........l........]................]........]................................ AINALNOI 
I Oot I 06 I 09 Լ 04 Il 09 Լ 09 I Oh Լ 04 I 02 I {3d 404 
A I 49/4 209 
I id MOY 
I ANNO) 
O4NI 
90३ Ց Կ Փ Տ Փ Փ ԵՓ ३ SS W ४ ३ ३ क ४ S S W N छ ३ ३ ४ W ४ ३ ४ ४ ३ ४ ३ के YV R W N W W W ४ ३ W ४ ७ के $ $ W 


१329 AB G3QTAONd NOTIYHNOINT 40 ANIIVIA 
क कक कक ՓՓ ३ ३ ७% ७ ७ क छ क क 40 


OdNI AO OL BINVIGI@CY YO 3)IAOY dO NOTANOIULNOS BIWINOD 
OT ORB. NO 8 0१ 3 Ց Ց. ७ ४ ४ ४ ७ ४ ७ ४ ४ ४ ३ ४ ७ ५४ 


जल 


եք 


है | աա | էշ 


| y մ "जे 


խան Ա 





(6 
Cat 
(१ 
(१ 
(2 
(Fi 
(८४ 


(6 
(२३ 
(¢ 
(9 
(2 
(Fi 
(Ch 


N 


a MU Կա x x Հժ աժ 


«աժ Հոթ Պօ Կ» աթ աթ աժ 


‘sbutAaesg Jo oddéAy Aq buraes Jo Areunung 


१ 
0 
60+3ç%ç2?n l° 
0 
0 
0 
8N+9SSFRAT° 


FJINVIMVA 


4 


sburaes 3० उ569 Aq buraes Jo Kapuuns !3arjJəuəg upaBoad əoSupeqsrss 


20302 


604+ 30S0RT° 
0 
9093३/९$०१" 


JINVIMVA 


eo ofa 


AAZO SENET 
0 


ROEL* ZIRE 


॥३0० OLG 


z JO T obeg 


9T# 340d53J - Burtaes 34502 Pury3eaado Tenuuy 


O Ə D 2 


o000°00SST 
0 
6009" 899 


NV 3W 


0 HIHIN 
0 3BVAVRTLS9 LON 
0 O VANNY NI SONTAYS 
0 YW TYNNNW NI SONTAYS 
0000*nontf Jwti JNO NIT SONTAYS 
0 

00000०0००१६ 

uns VIAVT INTIVA 


` ONTAVS 40 3dA 


‘ATyouog wexrhorg 9S9IULISTSS 


"9 
"h 
22 


նե: 


լ 


AYS (603 BIYd38 80 NNTLINNLENNI JHTE INO 


SNnOTIviaindd d ANS 


30 


Կու ա էԼ हे 3 p 3 Q 


gr Sanpra 


Sh տ §$9Sv) viol 

0 0 0 ՎՅԵԻԼԸ °G AYSAAL 

0 0 0 3T8vHILIS3 ION "հ AYSAAL 

PESE OCOTI £999°999¢ 4 0N00*000lh N “AYNANNY NT  SONTAYS °$ AvSdAl 

0 0 0 . v TVANNY NI SONTAYS *2 AYSdAl 

0 0 0 ՀԱԼԱ INM NT SOINTAVS “I AVSdAl 

0 0 0 0 AYSdAl 

haf heen ՒԼ ԷՒ Tes 0000°000th NOTLVINdNd JHLINI 304 
A30 ALS NYJW ԽՈՏ VWAVT INIYA 3009 JIAVIAVA 

ANTAYS 30 3dAl AVGdAL AR NNOO N3NONN 
ONTAYS ISD) INTLYMIAN AvNNNY AVSOY JIQVIAVA NOTHILI a9 
»» €6NOQOTLvATANd Nd ANS 39 ԿՈ Կ ՉՏ ՅՕ क क क क क कक कक क क क क क क 
9T# 3a0d5y - DBurAeS 3S0D atedaoy «9 սօլգջույչջճսօյ տալն SUO 


8p sandra 


५१ 9३०) ՂՀՀԼՈԼ 
AVSdA] 
AVSdAl 
AVEdALl 
AVOdAl 
AVSadAl 
AYSdAl 
NOJLVINAOA JNIINI HOY 
300) JVAVIAVA 

AVSdAl AS NNMOG N3NONA 

AYSHLI 3I8YINYvA NOIY3LIM) 


UGS 





(७ 
(२! 
($ 
(१ 
(२ 
(et 
(Sh 


(6 
| 
(१ 
(9 
(? 
(eT 
(Sh 


M 


wf “ad "ad "ed "ad "ad աժ 


աօ Կաթ «Ք «ա «աթ Պա Փա 


0 0 
ट९।+३८११९११° 50६७१५१0०७१ 
0 0 

0 0 

0 0 

0 0 

234 306nnt° TSST’nntaRf 


3 3NYIMVA A30 als 


‘sbutaes Jo ՅՎճն Aq Hutaeg JO Areuuns ‘4tyousq wesahorzg DJOURISTSSY 


0 Ո 
0 0 
0 0 
RO+INLYLI® QLR0°HN2n 
0 0 
0 0 
CHZ9°HTHTLAM Sane*Lnee2 
ԶԿ ԼԱՈ A3O 048 


տ տ տ տ տ տ տ տա տ a S N M I լ y १ n Վ D d 4 Ո S 4 Ո 


0 
ECEE°OLOLHE 
0 
0 
0 
0 
७990" 2५५९७ 


Z 
G 


SNO TIivarnanhaans 4 Ü 


JO z ՅԾԵզ 


१ Jı0day - 350) 308[L0Aaq4 


Jo adAy Ag Butaes Jo Areuung ’4TJ9uSq weaßolg 9SDULISTSSY 


8b जया» व 


sh e 8382) ALO! 
0 N3HIO °S AVSadAl 
0000’nneangin JVAVWIIS3 ION "Ե AVSdAL 
0 Ը IYNNNY NI SONIAYS "ճ AVSdAl 
0 VN TANNY NI SONTAVS "շ AYSdAl 
0 INTL INN NT SAINTAVS I AVSdAl 
0 0 AVEdAL 
0000*nnan9In NOTLYIWNdNd FINTINI 8093 


WAS TIRVI ՀՈՊ 3003 


INIAYS JN JdAl 
ISO) $33f0kd 


3IAYINVA 


AYSdAl AQ NMDO N3x0H8 


ISIFUA 
N D I l a ) ԿԴՔՇՏ 3 OQ տ տ տ տ տ का काका ७७ տ տ տ . a 


JIAVIMVA NOIHILINI 


9T# IX0d3y - Hutaeg 4502 Յծսծսթգալելը Tenuuy 


£££*226n 


नब OOM OF “० 


TIE” 969 


NY H 


gp Sandra 


Sh s SISY) With 
0 MSHIN °S AYSdAl 
0 3)9vwTISRA ION "հ AYSAAl 
0 O IWOANNY “ET  SINIAVS "ճ AYSdA]l 
0000" %१६४५७?2 YW FTvNMNhkY NT SONJAYS "२ AYSAAl 
0 ITIL IND NT £ONTAVS "lÍ ॥ १५००५ | 
0 0 AYSdAl 
0१000" १६५७९ nNIAYINANd TIN 809 
uns VAP VT INNA 300) IVAVIAVA 
ANTAYS Վո JdAl AWEdGA] AN NMDO Կ3»ՈՒՑ 
१४] ॥9$ ISNI JIINYNFJINT VW TYNNNY AYSHY JIAVINVA NOTNALIND 


NAOT lad TIT NJC JQ 2.0. 


770 





(6 
Cat 
(4 
(Գ 
(2 
(ei 
(Sh 


(ç 
(9 
(ne 
co! 
(Sh 


Sf waf sf x x sf w 


աժ Պաշ `ՀԺժ աժ աժ 


0 

60492820%° 
O0t4392S2ET° 
60+300Sf2° 
60+399p5ç ° 
0 

604399TT02° 


SINVIAVA 


Jo }tjJousg wnutTuTW 


CoN 


0+35७525° 


JINVIMVA 


0 

Ond2°ontot 
0£t2"non9£ 
।९००१०७२६५! 
१।५५*०१७५११ 
0 

TLEt sett 


१३० 028 


e= BNO TILVTAd Od ANS 30 


0 
Ո 
0 
0 
Q 


0S2°eS2L 


A39 Gis 


== GNO TLvtndadndans 


0 

ԼՉՉԳ" ԶԼԻԵծ 

£999°66SNE 
cLFE°otert 
000००००१११ 
0 

LLE ०0०९८ 


NY3W 


8T+ Zaodsey - 4Izyeueq Jo 9dAL 
Aq s}tyoueg poyenteag jo Axzewuuns ‘3 tTJaeueq werhorg S9ueJsTssy 06 SANBTA 


0 Yy3IHLO 
0000°Q0LET? 378ywris3 LON 
0000°662¢€0t O IYVNNNY NI SONTAVES 
0000°0%628 YH IYNNNY NT SONTAYS 
0000*009£2 Anti INM NT SOINIAVS 
0 

0000n*L¿ton2£ 

ԽՈՏ J3AYI INTVA 


"հ 
०६ 
"2 


ONTAVS 40 IdAl 
LY 43 3N38 


NOT aTe a 


0 


८T# 324043 - sysonbey 7284 फा ४» 
poqeutysq ‘3ZtTyousegq werzforg 9D9UeISTSSY 


0000°00NS2 
onngn°*onost 
ANNO OGL? 
0000* 052 

ISSS SGAS 


NV 3H 


0000°000SL 
0000° 00002! 
0000° 00099 
0000°00S2 
0000°00S€92 


ԽՈՏ ՂՅՔԿՂ JNA 


43४36 40 JLVWIIS 


°S 


ան 
ne 


3 


LIIINIR 


30 NOT }adTNMISI0 


sh e $38v9 Iviol 
AVGdAl 
AVSdAl 
११9०४ 
॥ १४०७ ३/ | 
AVSdAl 
AYSdAL 
NOTAVANIOA IYTINI YOS 
3009 JIGVINVA 

AVSdAl AB NMOO NIYONR 

i JING JIAVİNYA NOINAILIVI 
6p Sandra 
Gh = §3Sv) Will 

¡STILINA 
ISJFL4N8 
IS 3L4NQ 
ISJZi4NeA 
NOTLVINdNd INIINI 803 

3909 JIAYIMYA 

AS3I4N9 AQ NMOO NINNMMA 
L INH JIAVINVA NOTHILING 


J. D 





W8T# 3३2००३ 
- sSYTJaUusg payenTeAg A0OJF SOTISTIJEIS 'ITJOUOg weibolId SOUPAISTSSY T 


ç Sanbta 
999" 665nf çççç*°Qleçi 000 °O00RII ԼԳԳԳ"օԼիծ NV 3N 
६ շաղ 2 den [ ann Ե «ԿԳ ANNY9 
J LISANS 


(37S IvHL 3N LISANS vy Nid JONVN INVIIAINGTIS 
ԼՏՅԱԿՈԿՑ ՀԿ) Կոլ ՀԿՈպԿ AA AM3IITO IvH} SNYIN JAVH HOTHM JN NIYAJ AN "SANDMAN 40 StasHns) Sj3ISANS SNOININNHOY 


CCCCRON/T + CI)N/T) "= SINTNOS-NYIR=-SANDNI=-NTHLIF wv S°0)luH0S 


¥ SONYA 
ST (PUNY IW Mi CLONV QW JIAYAWOY NI] AISN AVIWNLIY INTVA Հալ 


"ՇՈՂ" HVINAVI INV S39NVH JANAY ՀԵ 
L७°¶ १५०९ 96°? 


< VIA37 060° Հոլ UNI SIINYN 
3«ՈՈ3ՀԴՈԿՒ:Ց ՏՂՈՀ»"»ԽԻԾԿՅՀԿԽ«ԱՀԴՈՈԼՏ 


iSJ3i JONVH II4TLINW 


THING |) 399 ATFTAVTAVA 
= =< = =“ = =“ = = ewe =“ = =“ ww ew = = = = = = = A V M INN <= = < = = = = =< = = “< = = = = = = = = = = = < = = = 
QoeQQg'hocosnelqoqQ २२ Vint 
267b°SSSFS6L92 L999* M2GLTTT69S 6l San 89 NIHI TI" 
११! " [06 ° í «06 Լ"ԳԵեՏՔՉԻծՈՉ 62N2°OHNGIERASI ६ SANON ԿՀՀԻԼՀՈ 
"Զում 4 NILIY} 4 6३/४३/१११५ १३११७ SANVAMS ՎՈ uns ०५०0 IDANOS 
JINVIAVA IN STSAIvNV 
ANTAYS 4M JdAl AYSdAL AA 
LTdIngA Ldn) JVAVTUVA 


eeeeeeeeeiA YX IN & ०७ « « = 


Լ 





(२ 
(ot 
(6४ 
(6 
մն 


N 


we? पक्की wf w աժ 


Aq qarjəuəgq 


0 
0 
0 
0 
9 


JINVIMVA 


0१30७०२१" 


m 


UruTeuoy WNAUTUTW PoFeuTtIsg ’I3TJousg weıborg SDUBISTSSY ք. ՅՀՎոԲյյ 


“OOD 


११८१0०८५ 


A30 QIS 


0000°000S2 
१00००००५१ 
000१*०५८2 
0०००४१५२ 
LSR2°HtTAh 


NYIW 


028 ԴՂԾՎԱՅյ - SOAFeUTISY ITyouog 


0000°0000S 


ՌՈ000"Ը00094 
0000"0օ(Լեճ | 


0000°0S22 


0000*000L€£ 


nos 


(१ 
(AT 


Հա ԿՓ ԿՓ ԿՔ աթ աշ աթ աթ աժ 


0 
0 
0 
HOF WHOHOT® 
60ՅՀՅե6ՈԼԼ" 
0 
60+3६९5५0/६५ 
604328829° 
60+३१।१0०२१ 


JINVIMVA 


0 
0 
0 
E6AT°OSOh 
t€00°HISGOI 
0 
Lhez° onze} 
Լ890"9(062 
TLETT °ERtys 


A30 OLIS 


- 90000 4Tyeueq Aq sqtjyousgq Jo Areunmg 


0 

0 
0000*00062 
$55 "L9LS 
ETFS’ SE HL 
0000°6F2 
2१११०११८१ 
[nil’6nn?22 
8८८६"002/ 


SNOTivtnandanese 40 


0 

0 
0000"00096 
0000*0०७२।१ 
0000 RIntTE 
0000°6€2 
0000°2262S 


O000°RHTLSS 
0000°LIOHe¢ 


HAS 


B NOT IV WNdOd ane 4 0 


LIS3IN3IR IN JlVWTIS3 


0 "= S35४) rij 

०५ IS7i4NQ 

"հ AÄSIL4NG 

"4 IS3LYINA 

०? 4534 4५7 

NOTLVINdNd ՀԵԼԼԿ ԿՈՎ 

V9IAVT IN TWA 3003 3JIVYOVIBVA 
IASIL4N9 AO NMOO N3NONMNA 
L743N3A I dng 2Պ7 47 NOIMILINI 


ՎՈԼԺ ԼԿՉԶՄ 6ՓՅ0Օ 


‘ATjousgq werborg əoupqasrss 


W fQ3N30T8NN9 
ANN 'AINNY AA 
83 LON ’'NJyyI| 
լ *'03830I6N03 
INN *!Q3NNY d 


WHII99 “/|४३५९४॥ 


67# F70dey 
CG eanbty 


Gh 8 8३९६७) IvlOl 
LI49N39 OWN "հլ 0OILING 
O4NI Iv83n39 "१! 0094 J4N6 
NOTIIV ०? O0OJILJNR 
NOTLIv "9 QOJLINA 
NOIIIN °L OOILYINO 
nNOILIV "6 00ILINR 
nNOILIV "Հ OOILIYNR 
NOTLIV ալ 0024 4|५ 
nNOILYINAOA 38ILN3 NOY 
INVA 3009 3IVAVINVA 

LI4 9N38 0220 ५५४ (पि १५१ए १३००४४ 

LI 43N38 L ING 3IVIVIBVA NOINXJLIİNI 


Կոլա Ր 624 7 0 


SP 





TZ# 330dəy - 3T39U9%H pue 
SIEUTISI ITJ9USE յօ սօտլմծմաօջ "լս werzhoAg DOUBISTSSY pS sanhty 


arool 6" 8 0°02 nen? 2° 2 6" Օհ Iy LOL 
०१ p 6 6 | हट ԿԱՈՂՈՉ 
AAA AA asf [०००७८७७ ७ ७» ७ ७ "ूँ a 
Lee PE t n I[ 0 | e? I 
MTD oe TUL I 0 t 0 I S°h I 
("9 lore SS IO SI On T 0 I “°$ լ 
ç I Í ı A T दी I 9 | | Is 
[manmnnun[annnnnnn[nunnnnnn[ennnnnne [onnnnunn] 
Ir «իժ [ ('Գ9Գ I 0 I O I 6'4 T 
| “OTe ST १०$६$ J ն T 0 I e’eı I 
RLI TI S°?I T ५९"९/६ I 0 ae T १200 ] 
8 ԼԵ | [ $ T 0 I N In Ih 


fecoccconeaj]snconano[ rara rana ara arar lo 

I oon T 6° Լ ՔԱ | 2°? I "02 T 

I 0209 հ"հԵ է Բ"Ք. լ OfONT [ 8१0१ լ 
९०९० I 4" 9 LALA ON t so | Շիր t S'/€ I 


ne T 2 I հ T ք T I I 6 I $ 
प(_[००७००००७ ०७०७ 'ूं ७ ७ ७ ५ ७ ७ क ७ | ००० ७ ७ ७ ० ७ ७ एूँ ० ००० ०७ ७ ७ ७० ७ हू ००७ ३ ७ ० ० ७ ७ | ०» 
Լ 0 Լ 2°2 Լ 2°? T | ee I 
I n I Pl I ("լլ | A I १०५१ լ 
२१२९ I 0 [Ot տի I O I 0208 I 
nt | ०) y | T | T 0 [ ४ il e 
J town none | ७ ० ७ ८ 9० ७ ० ७ हूँ ० ०७३० ० ७ ७ क क हैँ ००० ० ७ ७ ० ७ क० |ूँ क ७ क १७ ७ व ७ ७ ५ूँ ०० ० ७७ ० ०७ 2० ३७ IS 3L4N6 
I ५ Ton I ç Mr: լ ն Ւ Lod int 
Wink լ (32 999 
“NN [ Jd "ՈԿ 
լ INNO) 
LANA 


» Բ Ք Ք क NN ४ ॐ कै Y yy ३ Ե Ի ही छं कै कै कै कै क क ३ yy W के की के के Ք« ३१ ॐ १ ३ W VV W Y ५ YN VY 


iT43N39 LINA AW LT43N9H IN Alywfis}3 LS 3i4N9 
x Mm կ ՎՈ աաա թՏ եռ O 4 | ՇՏ HN.) row vex eo ՔԱ Կ տ Կ Կ ԺԳ क 9 Մ 


_. / 4 





BO. 


ge) 


REFERENCES 


Drucker, Peter F., Management: Tasks, Responsibilities, 
Practices, Harper & Row, Publishers, Inc., 0974 , 
pp. 45-46. 


Hach, HNC ara Wilson. W.; A Survey of Industrial 
Research and Development Budgeting, Effort Selection and Development Budgeting, Effort Selection 
and Evaluation, thesis prepared for the Naval Facili- 
ties Engineering Command, September l976. 


Naval Facilities Engineering Command, NAVFAC P-44l, 
Command Management Plan, FY l977. 


Sri Engineering Laboratory Instruction 5450.2P, 
"Organizational Chart and Functional Statement of 
the Civil Engineering Laboratory," January 7, /9775 


Naval Facilities Engineering Command Instruction 3960.2A, 
"Research Development, Test and Evaluation (RDT&E) 
Assistance; availability of," April l, l976. 


Jolly, J. A. and Creighton, J. W., FESO Project Effective- 
ness Profile: Summary and Analysis of l972 Question Profile: Summary and Analysis of 2972 Question- 


naire Results, paper prepared for the Naval Facilities 
Engineering Command, October 2973. 


Jolly, J. A. and Creighton, J. W., et al, FESO Effective- 
ness Profile: Summary and Analysis of TS Question of I973 Question- 
nalre Results, paper prepared for the Naval Facilities 


Engineering Command, September 2974. 


Hendrickson, J. A. and Fisher, W. G. Jr., An Evaluation of 
the Effectiveness of a Research Organization's — 
Mechanism for Transferring Technical information to for Transferring Technical Information to 
Applied End Use, thesis prepared for the Naval Facili- 
ties Engineering Command, December l974. 


Suess, K. M. and Thaler, J. F., Demonstration of the Feas- 
ibility of Automating the Information System of a Information System of a 
Small Service Organization Using a Generalized Compu Service Organization Using a Generalized Compu- 
ter Software Package Software Package, a thesis prepared for the Naval 


Facilities Engineering Command, September 2976. 


Nie, Norman H. Hull, C.Hadlai; Jenkins, Jean C.; 
Steinbrenner, Karin; and Bent, Dale H., Statistical 
SS SSS Ee for the Social Sciences, New York: McGraw- 


Spiegel, Murray R., Schaum's Outline of Theory and Prob- 
lems in Probability and Statistics, McGraw-Hill Book 
Company, २975. 


J 75 





HORN Burlington, R. R. and May, D. C. Jr., Handbook of Proba- 
bility and Statistics with Tables, McGraw-Hill Book 
Company, 4970. 


l3. Freeman, H. A.; Friedman, Milton; Mosteller, Frederick; 
and Wallis, W. Allen, Sampling Inspection, McGraw- 
Hill Book Company, Inc., 2948. 


l4. Naval Facilities Engineering Command, NAVFAC P-442, 
Economic Analysis Handbook, June 2975. 


L 





EO 


INITIAL DISTRIBUTION LIST 


Defense Documentation Center 
Cameron Station 
Afexandria, Virginia 223.04 


Library, Code 0l42 
Naval Postgraduate School 
Monterey, California 93940 


Department Chairman, Code 54 
Department of Administrative Sciences 
Naval Postgraduate School 

Monterey, California 93940 


Dean of Research, ९००९ 022 
Naval Postgraduate School 
Monterey, California 93940 


Professor J. W. Creighton, Code 54Cf 
Department of Administrative Sciences 
Naval Postgraduate School 

Monterey, California 93940 


Professor J. A. Jolly 

School of Business and Public Administration 
California State University at Sacramento 
Sacramento, Edlifornıae95379 


Mr. Eugene H. Early, Code L03C 
Civil Engineering Laboratory 

Naval Construction Battalion Center 
Port Hueneme, California 93043 


CDR R. P. Cope, Code 03 
NAVFACENGCOM 

SHON HEV | | Street 
Alexandria, Virginia 22332 


LCDR A. A.-Arcuni, Code ԱՅՆ 
NAVFACENGCOM 

200 Stovall Street 
Alexandria, Virginia 22332 


Mee tem Gamo, Code 032€ 
NAVFACENGCOM 

200 Stovall Street 
Alesranerıa, Virgınlıa 222332 


277 


No. 


Copies 
2 





ll. 


2 


L3. 


is. 


ts. 


26; 


T7. 


TB, 


i 


2: 


Mr. Milon Essoglov, Code 032A 
NAVFACENGCOM 

200 Stovall Street 
Alexandria, Virginia 22332 


Mr. T. Rohrer, Code 0320. 
NAVFACENGCOM 

200 Stovall Street 
Alexandria, Virginia 22332 


Mr. Steve Hurley, Code 032D 
NAVFACENGCOM 

200 Stovall Street 
Alexandria, Virginia 22332 


Captain Norman W. Peterson, CEC, USN 
Officer in Charge 

Civil Engineering Laboratory 

Naval Construction Battalion Center 
Port Hueneme, California 93043 


CDR R. Lowell Preston 

Assistant Officer in Charge 

Civil Engineering Laboratory 

Naval Construction Battalion Center 
Port Hueneme, California 93043 


Mr. William F. Burkart 

Technical Director 

Civil Engineering Laboratory 

Naval Construction Battalion Center 
Port Hueneme, California 93043 


Technical Library, Code LO8A 

Civil Engineering Laboratory 

Naval Construction Battalion Center 
Port Hueneme, California 93043 


Mr. E. F. Humm, Code 2028 
Northern Division 

NAVFACENGCOM 

Philadelphia, Pennsylvania 9222 


Mr. David W. Gurganus, Code 09P2 
Atlantwe Division 

NAVFACENGCOM 

Morfelk, Virginia 2354. 


Mr. E. A. Byrd, Code 90 

Southern Division 

NAVFACENGCOM 

P.O. Box 20068 

Charleston, South Carolina 294ll 


J 75 





Zi. 


22: 


23. 


24. 


5. 


26 


2l 


Dar 


29. 


20. 


Srl 


32. 


Mr. Walter L. Cowell, Code 2077 
Western Division 

NAVFACENGCOM 

P.O. BOX 727 

San Bruno, California 94066 


Mr. William C. L. Young, Code 402 
Pacific Division 

NAVFACENGCOM 

FPO San Francisco 96620 


Technical Library 

Civil Engineering Corps Officers School 
Naval Construction Battalion Center 
Port Hueneme, California 93043 


LCDR Jack E. Hendrickson, CEC, USN 
COMCBLANT DET 
FPO New York 0952] 


LCDR William G. Fisher, CEC, USN 
NAVFAC Contracts Bethesda 

200 Stovell Street 

Alexandria, Virginia 22332 


Naval Aviation Executive Institute 
Department of the Navy 

Naval Air Systems Command 
Washington, DC - -2036 


Mr. Kenneth M. Suess 

Management Systems Development Office (Code 220) 
Naval Air Station, North Island 

San Diego, California १92235 


Mr. James F. Thaler, Code OP-43 
Naval Air Propulsion Test Center 
Trenton, New Jersey 08628 


Mr. Perry B. Newton, CP5-866, Code 03T2 
Naval Material Command 
Washington, DC 20360 


Mr. Howard C. Fish 

National Parachute Test Range 
Technical Director 

El Centro, California 92243 


Dr. Stephen Laner 
BI Scenic 
Berkeley, California 94709 


Mr. Joseph W. Wilson, Code lo322 


Pacific Missile Test Center 
Point Mugu, California 93042 


T73 





5. 


34. 


55- 


36. 


Di]. 


23. 


29. 


Dr. P. A. Phelps 

Bechtel Corporation 

50 Beale Street 

San Francisco, California 94lI05 


ZET V. M, SKfinak, CEC, USN, Code! 032 
Naval Material Command 
Washington, DC 20360 


Eo Maurice K. Kurtz 
U.S. Army Engineering Topographic Laboratory 
Port Belvoir; Virginia 22060 


Western Region 

Technology Transfer Resource Center 
Naval Postgraduate School 

Monterey, California 93940 


Mr. Allan A. Sjoholm, Code 20L 
Navy Personnel Research aná Development Center 
San Diego, California 272752 


CRUSK 

Institute for Social Research 
University of Michican 

P.O. Box ]\248 

Ann Arbor, Michigan 48206 


Denver Researcn 


University of Denver 
Denver, Colorado 80220 


TOO 








Thesis f ie TE ह 

| El2724 Early 

Suk The measurement of 
efforts to enhance the 
effectiveness of NAV FAC 
RDTEE investments. 





4 
thes६I724 


The measurement of efforts to enhance th 















































| 











մի 


3 2768 00 90268 7 
DUDLEY KNOX LIBRARY 









































































































































